Post #906661

2026-04-02 20:08 UTC

I&#39;m analyzing Medicare data -- my first real experience with a large dataset, where the number of observations of interest to me is in the millions. We have repeated measures/clusters to worry about, each ranging from 2 to 10 observations, give or take. I&#39;m struggling with performance issues in pretty much every approach I take to this dataset. One outcome of interest is a proportion. zoib is painfully slow, even when I take a (stratified) random sample of 2% of rows -- in an hour it&#39;s only 4% done fitting my null model. Boundary values (0,1) are common in the data, ruling out &quot;transform and just do lmer.&quot; What general tools are available for modeling bigger datasets in R? Because of data privacy agreements I&#39;m required to do all of the computing on-prem, so unfortunately I don&#39;t know that I can take advantage of high throughput computing on other servers, if it were even workable in this case. #rstats #lme4 #zoib

Replies (6)

@BRicker@fosstodon.org 2026-04-02 20:14

@emjonaitis a local DBMS instance with appropriate indexes defined and using the relevant R-dbms interface might be worth the hassle in this case? (Or pre-processing in DBMS to small extract files.)

Open ##2154508
@olyerickson@mastodon.social 2026-04-02 20:23

@emjonaitis Have you looked at https://hbs-rcs.github.io/large_data_in_R/

Open ##2154510
@erikjan@fosstodon.org 2026-04-02 20:37

@emjonaitis how many/what types of predictors do you have? If you really want to zoib that's just a really intense inference procedure. Options: do "standard" beta regression which is maybe a little more efficiently implemented with ML (rather than Bayesian) estimation? It commonly shrinks 0 and 1 a little bit towards 0.5 as a preprocessing step so it'll be a bit less exact but it may not matter at all with this much data Another option:

Open ##2154511
@erikjan@fosstodon.org 2026-04-02 20:47

@emjonaitis this is a stretch, but do you think BRMS / Stan with Laplace approximation will be any faster?

Open ##2154515
@Tourma@tech.lgbt 2026-04-03 22:32

@emjonaitis At firsti thought something like folding@hime or bionic, but protines have a lot fewer privacy issues than people. Try and convince your higher ups to buy you a used bitcoin farm?

Open ##2154521
@erc_bk@fosstodon.org 2026-04-03 23:14

@emjonaitis Doubt these package can be used for entire dataset, but here are some beta regression packages that handle clustering and have some optimizations included (e.g. C++, parallelization). Not sure about the 0s and 1s though. Anyway, they might worth checking out. betaregscale, https://evandeilton.github.io/betaregscale/ cobin, https://github.com/changwoo-lee/cobin glmmTMB, https://cran.r-project.org/web/packages/glmmTMB/index.html GLMMadaptive, https://drizopoulos.github.io/GLMMadaptive/

Open ##2154522