Post #1561762

2026-04-23 06:37 UTC

Confessional rant incoming; I spent a significant amount of the last two days messing around with Copilot (as recommended and encouraged by my employer), and the sum total is that I have been gaslit for about 6 hours and have learned very little of use. I really think I might post the logs somewhere, except they are very tedious and boring. Broadly; yesterday, I have a very large (~168gb per file) spatial data-cube I want to do some calculations on. Too large to fit in memory, and the processing involves time-lags which means I really need to look at each cell independently. I wanted to make an &quot;empty&quot; template copy of that dataset so I could place the calculated results into it, cell by cell, hopefully in a way that might work in parallel. Cue: several hours of: &quot;Do this...&quot; Yeah that didn&#39;t work because of X &quot;Oh you&#39;re totally right, very perceptive of you, you understand why that didn&#39;t work. You need to do THIS.&quot; Yeah that made things worse, now this is broken. &quot;Good catch! That never would have worked and here&#39;s why. You need to do this instead.&quot; ...Eventually it concluded, logically enough, that it might be useful instead to output a bunch of individual files, each representing the time-series for one cell. That way I can run each cell (or a group of them) in parallel, and then merge all those cell files back into a cube at the end. Great. Copilot said this was &quot;Recommended, typical, HPC-safe way to do this&quot;. Cool. Today; Okay, how do I use cdo, or gdal, or R, to merge all these thousands of single-cell files back into a data-cube that has the dimensions/metadata etc. of the input variables? Repeat of yesterday: &quot;Do this&quot; Yeah nope. &quot;Of course, that was a bad choice, do this instead&quot; Nah, it&#39;s fucked mate. &quot;Well obviously, but good news, all you need to do is this.&quot; Nah. &quot;Fair enough, but I have a simple solution, give this a try&quot;. ... The ultimate conclusion (which I don&#39;t know if I believe yet) is that no spatial tools like cdo or gdal are capable of really dealing with a &quot;raster&quot; that is just a single 1x1 cell, and if only I chunked things into anything larger than that, 2x2 or 1x1500, it would all be fine. I haven&#39;t tested this theory yet, and there are a few reasons it would not be a great way to do things, but I&#39;m done. This was really not productive, I&#39;m sure I churned through more computing resources having this conversation than my HPC processing of the actual data will use. I&#39;m sure some chucklehead will say &quot;Well it would have worked if you had used Claude&quot; or something, but that&#39;s kind of not the point. We are being pushed to use this *specific tool*, we are wasting a bunch of money on this *specific tool* and being told it will make our work easier and more efficient. It&#39;s bullshit.

Replies (11)

@jimbob@aus.social 2026-04-23 07:52

I will admit, this is probably all not helped by geospatial tools being [[extremely silly]] and NetCDF files also being [[extremely silly]]. I went through multiple iterations of: "Of course, you naturally need to store the dimensions as (lat, lon, time)" Okay but now this next step is failing. "Oh that's because the convention in that file format is (lon, lat, time), you had better change it to that!" Okay but now I'm getting this new error. "Of course you are! That was expected behaviour and only natural, because this tool I've recommended you use requires the time dimension first! Here's code to change the dimension order to (time, lon, lat). Guaranteed to work." Folks, it did not.

Open ##1645019
@Kels_316@aus.social 2026-04-23 06:39

@jimbob average user experience I think. Except for chuds that are getting it to organise their email folders or whatever. But actual work, nope no good.

Open ##1645020
@skua@mastodon.social 2026-04-23 06:49

@jimbob This calls for a global all-hands meeting in which you present and share what you've learnt about using AI, and how you've learnt it, in a half day webinar using Powerpoint AND an AI video overlay that makes you look and sound like Elon Musk. With enough breathless excitement from you about your "stunning breakthrough", your boss will authorise this. #AISlop

Open ##1645021
@bencourtice@aus.social 2026-04-23 07:54

@jimbob circa 2002 I contemplated a career in IT. I am definitely not cut out for it, I have made many dumb and self-destructive career moves but that was not one of them. Imagine dealing with this shit as your primary work. Is anyone cut out for it? At what point does being directed to use it become an OHS issue?

Open ##1645025
@doc@mastodon.social 2026-04-23 07:56

@jimbob having dealt with computations on large multidimensional arrays, a solution I found was to use the HDF5 format. This supports massive arrays of several data types (with optional compression), which can be lazy-loaded and lazy-written, so you can handle things slice-by-slice or line-by-line without overloading your memory. Be careful with selecting the right chunk size and dimensions optimized to your problem.

Open ##1645028
@eliocamp@mastodon.social 2026-04-23 08:05

@jimbob CDO will gladly produce 1x1 files to represent timeseries (e.g. from cdo fldmean), so I'm sceptical that it can't handle them afterwards. btw, cdo distgrid will chop up your files into several independent regions that you can process and then collect with cdo collgrid.

Open ##1645030
@ramblingsteve@floss.social 2026-04-23 08:17

@jimbob it's so great. i had to explain to a chatbot that it couldn't retrieve data because it hadn't created an API end point. eventually it agreed. apparently this is the new super productive way to work.

Open ##1645032
@PaulMcEwan@eigenmagic.net 2026-04-23 08:34

@jimbob We're required to utilise Copilot at my university. I'm refusing to do so and if I'm disciplined for such, I'm out the door, looking for another position. Maybe I'll become a barista.

Open ##1645034
@noodle@aus.social 2026-04-23 12:27

@jimbob Did it give you an variants on 'ah yes the classic blunder' or 'It looks like you've got the perfect storm'? Those were my favorites* when I went through the process of learning not to use it.

Open ##1645040
@allsumnull@mastodon.cloud 2026-04-23 10:39

@jimbob some colleagues are super bullish about Claude Code. I wish they were wrong, but there do seem to be some impressive examples.

Open ##1645042
@bakuninboys@aus.social 2026-04-23 22:29

@jimbob this has been my experience too. Extremely frustrating. It's so incongruent from the folks who see it as a true revolution (and a far cry from the world ending monster everyone's talking about). I just don't understand their experience. Are they lying? What's going on?

Open ##1645044