Landsat Earth for Orbiter (LEO)

martins

Orbiter Founder
Orbiter Founder
Joined
Mar 31, 2008
Messages
2,448
Reaction score
462
Points
83
Website
orbit.medphys.ucl.ac.uk
1. Introduction

As reported earlier, the next Orbiter release will include support for planet surface terrain. If you have missed it, here is the video I uploaded a while ago introducing a first glimpse of a preliminary terrain implementation:
Pretty good already, isn't it?


11. More things to try

At the moment, woodland areas look a bit dark and don't have much contrast. Apparently, this can be improved by mixing band 4 (near infrared) into the green band 2, because NIR has good contrast for vegetation. I want to try that next.

Also, I want to try histogram-matching the scenes against a global reference (the Visible Earth Blue Marble map that is the basis of the current Orbiter Earth texture). If I can colour-match the high-resolution Landsat maps against the VE map, then I could retain the latter for the lower resolution levels without a visible break when switching to high-resolution patches. Also, mapping against a global reference would ensure that the relative colour balance of the Landsat-generated map would remain consistent across large distances without drift or similar problems.


12. And finally, a plea for help

Writing the scripts for the image processing tasks described above took me about two weeks. But now the work only starts. So far, I have generated a single 16384 x 16384 tile covering 11.25 x 11.25 degrees (the central Europe tile used as an example in this article). The processing time for this tile took about 5 hours, not including the time for downloading the Landsat scenes (the tile is composed of about 100 scenes, each a download of about 300MB). My hard disk is filling up quickly, and I won't have the time, storage capacity and internet bandwidth to do the entire Earth. So I am asking you guys, as future beneficiaries of an improved Earth map, for help. Is anybody prepared to share in the processing work? If you are interested, this is what is required:

  1. A PC with 8GB RAM, running Windows-64 or Linux-64
  2. Matlab-64 (this is the killer, I suspect. I've written all processing scripts in Matlab. They are not very complex - if somebody wants to convert them to Python or similar, you are most welcome).
  3. Matlab must understand the geotiffread and geotiffwrite commands. This implies the Mapping toolbox, I think
  4. A utility to extract files from tar.gz files. Linux users are ok, Windows users can do it for example with 7-zip
  5. GDAL
  6. About 100GB HDD space during processing the scenes
  7. An internet connection that doesn't have problems with downloading ~30GB worth of Landsat data
  8. A way to upload the resulting map tile so I can pick it up.
Also (to cover all bases), by processing the scenes and sending me the result, you agree to pass the copyright to the tile to me. This is to avoid ending up with a map that has 500 different copyrights attached to it ;)

If there are any takers who fulfill all criteria and want to help, please let me know. I can then pass you my Matlab scripts and detailed informations. The more volunteers I can gather, the quicker we could end up with a global high-resolution map!
 

Epsilon

Interplanetary Road Pizza
Addon Developer
Tutorial Publisher
Donator
Joined
Nov 1, 2008
Messages
197
Reaction score
5
Points
16
Location
Seattle
Give me something that can run it in Linux without MatLab, and I can dedicate... probably 6 cores on a VM on my workstation and 4 cores on my fileserver to it, at least for a while.

Edit: Note, I also have probably about 2TB I can dedicate on the fileserver too. I can pass on FTP credentials, and I'll just need to know when it's been "picked up."
 
Last edited:

N_Molson

Addon Developer
Addon Developer
Donator
Joined
Mar 5, 2010
Messages
9,278
Reaction score
3,247
Points
203
Location
Toulouse
Ah, so the dream is becoming true. Thank you so much ! :hail:
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,398
Reaction score
578
Points
153
Location
Vienna
2. Matlab-64 (this is the killer, I suspect. I've written all processing scripts in Matlab. They are not very complex - if somebody wants to convert them to Python or similar, you are most welcome).
3. Matlab must understand the geotiffread and geotiffwrite commands. This implies the Mapping toolbox, I think

I think Matlab is indeed the killer here. I have a bit of experience in Python and could take a look at converting the Matlab scripts to it. I guess this would increase the amount of potential participants quite a bit.

There is certainly enough processor capacity to spare if the tools to access it are available to our community.

Looks fantastic, BTW! :thumbup:
 

SolarLiner

It's necessary, TARS.
Addon Developer
Joined
Jun 14, 2010
Messages
1,847
Reaction score
2
Points
0
Location
404 ROAD NOT FOUND
Fantastic job ! I would be very glad to participate if only I can download 30 Gb on a run without making the whole Internet in my town explode.
But I thought of something (crazy yes, but what not?): since your script can run in Linux, why not use a "processing server" that will participate in the automated task of making the tiles? That could be a bit expensive, and is really crazy I think, but I also think this will be the fastest way to get our Blue Marble mapped for the greatest simulator of all life!
 

Fabri91

Donator
Donator
Joined
Jun 2, 2008
Messages
2,178
Reaction score
233
Points
78
Location
Valmorea
Website
www.fabri91.eu
That's an astonishing improvement over the terrain we have now in Orbiter!

I'd be available to do some processing. The only thing is: what am I supposed to download/install regarding GDAL? I'm running 64-bit Windows.

Other than that, I'm all set.

EDIT: ah dammit, MATLAB. As others mentioned below, an Octave-compatible version of the scripts would make it possible for MANY more users to contribute.
 
Last edited:

MaverickSawyer

Acolyte of the Probe
Joined
Apr 11, 2011
Messages
3,919
Reaction score
5
Points
61
Location
Wichita
Oh. My. GOD.
G5hhP.jpg
 

BruceJohnJennerLawso

Dread Lord of the Idiots
Addon Developer
Joined
Apr 14, 2012
Messages
2,585
Reaction score
0
Points
36
This looks terrific. I am curious about the terrain formatting though. Will it be possible to define custom bodies terrain as a greyscale bitmap, or will that need to be replaced by a new method?

Also, what is the FPS impact of terrain as you go higher up over the planet? (ie more terrain in frame to be rendered)
 

martins

Orbiter Founder
Orbiter Founder
Joined
Mar 31, 2008
Messages
2,448
Reaction score
462
Points
83
Website
orbit.medphys.ucl.ac.uk
Thanks for the many offers of help! I really appreciate it. Give me a few more days to fix a few things with the code (I want to try the trick with the infrared band in the green channel). I'll then upload my Matlab codes, so it will become a bit clearer what I'm talking about. Then we can take it from there.
 

Hielor

Defender of Truth
Donator
Beta Tester
Joined
May 30, 2008
Messages
5,580
Reaction score
2
Points
0
Looks fantastic. I don't have MATLAB, but I'd definitely offer my CPU time to help out if there was a non-MATLAB script available.
 
Joined
Mar 23, 2008
Messages
165
Reaction score
0
Points
16
Face said:
Quote:
Originally Posted by martins View Post
2. Matlab-64 (this is the killer, I suspect. I've written all processing scripts in Matlab. They are not very complex - if somebody wants to convert them to Python or similar, you are most welcome).
3. Matlab must understand the geotiffread and geotiffwrite commands. This implies the Mapping toolbox, I think
I think Matlab is indeed the killer here. I have a bit of experience in Python and could take a look at converting the Matlab scripts to it. I guess this would increase the amount of potential participants quite a bit.

There is certainly enough processor capacity to spare if the tools to access it are available to our community.

Looks fantastic, BTW!

Looks fantastic. I don't have MATLAB, but I'd definitely offer my CPU time to help out if there was a non-MATLAB script available.

As above. As a 8-9 year user of Orbiter with no contribution other than the odd bug report and presence in the on-line community, I'd appreciate the opportunity to contribute a very small something back by processing. I would be dependent on non-MATLAB scripts, but can fulfil the other requirements.
 

N_Molson

Addon Developer
Addon Developer
Donator
Joined
Mar 5, 2010
Messages
9,278
Reaction score
3,247
Points
203
Location
Toulouse
Well, I'd also be glad to help if I knew how to. I don't fulfill all the requirements (4GB RAM only), and I have no idea of what Matlab or GDAL is, but it wouldn't worry me to let my CPU mix data all the night (or day). I have an optical fiber connection, though.

SolarLiner, maybe we could do something together, if you live near Toulouse. I could download the raw data for you, put it on a portable storage device (I don't have that though), then you mix the data, give it back to me and I finally can upload it ? Another thing I could suggest to you is to "infiltrate" the Paul Sabatier University of Science. Here they have really nasty DL/UL rates.

If an big online storage place was required to store and share the data, I'd be ok to contribute with several Euro-bucks. :tiphat:
 

Quick_Nick

Passed the Turing Test
Donator
Joined
Oct 20, 2007
Messages
4,088
Reaction score
204
Points
103
Location
Tucson, AZ
Ditch MATLAB and I'm here to help!
Windows 8.1 x64
8GB DDR3 RAM
6-core AMD CPU
700GB HDD space free
200Mb/s down, 100Mb/s up

Regarding MATLAB, I feel like GNU Octave would be the easiest to convert to. I look at Octave as a free and open MATLAB.
 

RisingFury

OBSP developer
Addon Developer
Joined
Aug 15, 2008
Messages
6,427
Reaction score
492
Points
173
Location
Among bits and Bytes...
Thanks for the many offers of help! I really appreciate it. Give me a few more days to fix a few things with the code (I want to try the trick with the infrared band in the green channel). I'll then upload my Matlab codes, so it will become a bit clearer what I'm talking about. Then we can take it from there.

Octave is supposed to be MATLAB compatible. If your code runs in Octave, I'll definitely give it a go.
 

SolarLiner

It's necessary, TARS.
Addon Developer
Joined
Jun 14, 2010
Messages
1,847
Reaction score
2
Points
0
Location
404 ROAD NOT FOUND
Well, I'd also be glad to help if I knew how to. I don't fulfill all the requirements (4GB RAM only), and I have no idea of what Matlab or GDAL is, but it wouldn't worry me to let my CPU mix data all the night (or day). I have an optical fiber connection, though.

SolarLiner, maybe we could do something together, if you live near Toulouse. I could download the raw data for you, put it on a portable storage device (I don't have that though), then you mix the data, give it back to me and I finally can upload it ? Another thing I could suggest to you is to "infiltrate" the Paul Sabatier University of Science. Here they have really nasty DL/UL rates.

If an big online storage place was required to store and share the data, I'd be ok to contribute with several Euro-bucks. :tiphat:

Definitely. I have a 1Tb HDD but it's not empty, still it got 200 Gb left so that should be fine for processed data. I'm all for a team work N_Molson!
You can send me a 64 Gb USB drive drive with the data on it that I'll return to you with processed data so you can upload.
 

Enjo

Mostly harmless
Addon Developer
Tutorial Publisher
Donator
Joined
Nov 25, 2007
Messages
1,665
Reaction score
13
Points
38
Location
Germany
Website
www.enderspace.de
Preferred Pronouns
Can't you smell my T levels?
The problem is not porting to Scilab or Octave, but finding geotiffread and geotiffwrite for these alternatives.

---------- Post added at 09:30 AM ---------- Previous post was at 06:50 AM ----------

Another thought that comes into my mind is that you can download 30-days trial of Matlab, do the job on full speed, and then forget about it.
 

Urwumpe

Not funny anymore
Addon Developer
Donator
Joined
Feb 6, 2008
Messages
37,605
Reaction score
2,326
Points
203
Location
Wolfsburg
Preferred Pronouns
Sire
My laptop should fit to the specs and is a bit idle since I mostly use my Surface RT in the evening, but no Matlab here.

Wouldn't these calculations be a good application for CUDA/OpenCL? Or even JOCL?

http://jogamp.org/jocl/www/

My machine specs:

i7-3610QM CPU
16 GB of RAM
1 TB of HDD
128 GB of SSD
Geforce 680

If storage is a problem, I also have a NAS...
 
Last edited:

Quick_Nick

Passed the Turing Test
Donator
Joined
Oct 20, 2007
Messages
4,088
Reaction score
204
Points
103
Location
Tucson, AZ
The problem is not porting to Scilab or Octave, but finding geotiffread and geotiffwrite for these alternatives.

A bit of googling shows open source versions of these functions for other languages. It should be feasible to at least script them, I would think.

And GeoTIFF specifications can also be found online if it helps.
 

martins

Orbiter Founder
Orbiter Founder
Joined
Mar 31, 2008
Messages
2,448
Reaction score
462
Points
83
Website
orbit.medphys.ucl.ac.uk
A few updates:
  • I decided in the end not to mix the NIR band into the green channel. I tried a 80% band 2 / 20% band 4 mix, but the result didn't look very good. Instead I'll just brighten up the overall green channel at the lower end a bit.
  • I managed to avoid all geotiff-related function calls except for generating unblended mosaics, which is only required for outputting intermediate results. It's not required for the "production code" unless you are interested in the different stages, e.g. for debugging.
  • I am looking into compiling the matlab scripts into standalone code. This would require downloading and installing the Matlab runtimes (free) on your part. For now, I'll only try Linux to see if I can get it to work at all.
 
Top