How to Process, Filter and Clean Yield Monitor Data
January 16th, 2008
Hi Everyone - long time no post, I realise.
Ok - so a bit of an update - I’ve spent the last little while learning a new language (VB.NET). It’s been a nice challenge, as most of my programming to date has been non-object oriented stuff - command line interfaces or totally embedded solutions usually written in ANSI C.
I’ve been up in North Queensland helping a company up there develop yield maps for their Sugar Cane harvesters (actually, I helped design the equipment that sits on the harvesters and does the yield monitoring, as well). The process of taking the raw data off the harvesters and converting it into yield maps is a fairly long and drawn out one. Firstly you need to filter the data (something I had been doing in Excel) to remove erroneous positions and obvious outliers.
Secondly you need to convert the latitudes and longitudes from the yield monitor into eastings and northings (A degree of latitude isn’t the same distance at every position on the earth) so that you can project the data onto the local industry GIS system (the projection used is called the Geodatic Datum of Australia - GDA Zone 55) which was a bit of a mission - I had to convert a long and convuluted mathematical process (about three pages of equations) into a neat little visual basic function.
Thirdly you need to tie the data back to actual weights to obtain a calibration for each machine (had a hybrid Excel / Access database for that job) and fourthly you need to have a way to automagically recalibrate the harvester as the season progresses and the pressures that the yield monitor uses to monitor cane throughput start to change (mostly they go up - wear and tear on the harvester).
I was planning to teach my clients how to do this all manually, but, since I had a hankering to learn a new language and wanted to make life easier for them, I asked a famous Googler what object oriented language he would use. He suggested visual basic, which suited me fine because it turns out that it’s quite similar to C in many ways.
So.. now.. a month later I know a new language and have a great
little humungous larger and more comprehensive than I ever imagined software package.
The package condenses what took me about 7 or 8 days to do using excel spreadsheets etc into an algorithm - essentially distilling it down into a 5 minute batch process. The danger is that it will look TOO easy and my clients won’t appreciate the effort that’s gone into it - but, meh, the satisfaction of a job well done was worth the extra effort.
Once the yield data has been processed by the software, it then connects into a GIS system which converts the point yields into a smooth surface - a yield map. The one I use alot is called ArcGIS, and it makes life rather easy. Specifically, it uses a function called ‘Inverse Distance Weighted (IDW) Interpolation’ to draw the map. Basically, for every recorded point from the datalogger, it looks at 8-12 other nearby points and uses them to ‘guess’ what the yield may be between the points - and hence comes up with a nice smooth yield map.
That process can take a long time - for a years worth of data for 23 harvesters in Australia the process of calculating the IDW maps took me about 3 days with my Opteron 165 based system - which leads me into my next post - about my new computer (feeling very Geeky here).
Entry Filed under: General Discussion