User Tools

Site Tools


en:programming_tasks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
en:programming_tasks [2013/08/06 10:14] – [Test] kakliken:programming_tasks [Unknown date] (current) – external edit (Unknown date) 127.0.0.1
Line 1: Line 1:
-====== ESA Summer of Code in Space 2013 ======+====== MLAB software development tasks ======
  
 +===== Distributed network measurement system =====
  
-===== Ideas =====+The network is intened for collection and distribution of measurements recorded on amateur volunteer stations. The resulting data are publicly available for analysis. 
  
-   - Extend open-source toolkit which can be used as an substitute for closed source programs. +{{ :univerzalni_uloziste.jpg?direct&500 |}}
-   - Design and implement reliable algorithms for distinguishing different types of meteors.+
  
-==== Open-source meteor detection toolkit ==== +====  Considered data inputs (network nodes) ====
-=== Brief explanation ===+
  
-[[en:rmds|RMDS01A]] is a SDR receiver system designed for radio detection of meteor trailsIt's advantage over other designs is a high performance software defined receiver and completely open-source designwhich allows for advanced signal processing of radio imagesThe primary setup we used included non open-source software like Spectrumlab and HDSDRWe therefore developed completely open-source alternative.+  * Meteorological data from stations [[cs:aws|AWS01B]] 
 +  * Radio meteor detection stations [[cs:rmds|RMDS01A]] - [[cs:programming_tasks#radio_observer| Radio Observer]] 
 +  * Video meteor detection stations [[cs:vmds|VMDS01A]] 
 +  * Data reported by accidental observers - [[http://wiki.bolidozor.cz/doku.php?id=cs:meteor-observer|Meteor Observer]] 
 +  * Seismic / infrasonic measurements (meteor impacts, explosions, ...
 +  * Radio telescopes  
 +  * Parallel observations from several astronomy sites 
 +  * Geomagnetic measurements 
 +  * SID monitors 
 +  * FRB monitors 
 +  * GRB monitors 
 +  * Network for lightning detection 
 +  * Measurement and coordination network nodes [[cs:abl|Automatických balónových sond]] 
 +  * Nodes acting as data storage, or performing some calculations on the data (data mining servers)
  
-== Software for RMDS01A ==+Each node as part of a network will be operated by the observational application (eg, [[en:radio-observer]], Meteor Observer, Visual Observer, etc.) in cooperation with  [[en:station-supervisor|Station-supervisor]] which manages the station verify its work. Another daemon should therefore have an interface or file system in which data measuring application should store data and then solve optimal data distribution between data servers.
  
-[[en:RMDS]]+====  System Properties  ====
  
-Most important pieces of software are doneWhat is needed is to glue them together to cooperate flawlessly +  - Measuring network should be able to send  short informational messages with low data volume as alert of special eventsFor example, in the case of Fireball Network a bolide detection which is expected to impact to the earthIn this case we need to measure other data on the atmosphere, such as wind direction at different heights
-For data acquisition we created program [[https://github.com/nnen/waterfall|waterfall]] which reads input from SDR receiver and generates both FITS raw files and jpeg previews of these files for interactive visual inspection.  +  - The station should be configurable through web interface on the central measuring project server. (Similary to [[http://boincstats.com/cz/bam/|Boinc BAM!]]
-For data synchronization with other nodes set of [[http://www.mlab.cz/WebSVN/listing.php?repname=MLAB&path=%2FDesigns%2FMeasuring_instruments%2FRMDS01A%2FSW%2FTools%2F#_Designs_Measuring_instruments_RMDS01A_SW_Tools_|bash scripts]] is used. Basic HTML5 interface was developed for live inspection of the received stream by many concurrent users over the internet+  - Central data data should have an API that would allow programming of other applications that use measured data (for live outputs in observatories and planetariums, or for educational purposes) 
 +  - An important parameter of the network is that data from individual stations were available at one time (with the least time scatter) in order to perform live calculations on the inter-station data.
  
-Some screenshots from [[https://github.com/nnen/waterfall|waterfall]] development: 
-{{:cs:sdr:snapshot_vlak_1368122931.fits.jpeg?600|}} 
-{{:en:varhany.fits.jpeg?600|}} 
  
-==Proposed projects:== +=== Typical station design ===
-  - **Difficulty**: //medium// **Importance**: //high//. GUI to glue all above mentioned software together to provide solid user experience and ease of use.  +
-  - **Difficulty**: //easy// **Importance**: //medium//. Finish the HTML5 GUI for watching live streams, implement interactive elements (like LO frequency changing). Optimize for use with mobile devices (tablets, smartphones). +
-  - **Difficulty**: //hard// **Importance**: //medium//. Implement algorithms for trajectory calculation of meteors. We will provide all algorithms to be implemented. Some very basic example: http://meteor.uwo.ca/research/radar/cmor_intro.html +
-  - **Difficulty**: //medium// **Importance**: //medium. Design and implement robust algorithms for processing the signal. The recommendations are to use one of: [[http://root.cern.ch/drupal/|ROOT]], [[http://www.gnu.org/software/octave/|GNU Octave]], [[http://www.r-project.org/|R]].+
  
-//+{{ :cs:designs:supervising_system.png?direct |}}
  
 +Image above illustrates an typical station connected to the distributed measurement system. The measuring station itself consist several subsystems. 
  
-=== Expected results ===+== Local presentation node ==
  
-Single package which will be able to install software needed for viewing, detection, collection and processing of meteor signals.+A PC system located at same site where the measuring station operates but generally in another room or building.  The purpose of presentation node is displaying of measuring data in an interactive form which is attractive for visitors or should be used to demonstrate principle of measurement.
  
-=== Knowledge prerequisites ===+== Measuring node ==
  
-  * Knowledge of programming languages necessary to implement the solution (C/C++ preferred for performance reasons). +This is part which performs the measurement and it is responsible to data makingMeasuring node has several minor subsystems.
-  * Ability to use/learn to use basic development tools (IDE, version control software, issue tracking systems). +
-  * Ability to learn and use third-party libraries necessary to implement the solution (CFITSIO, libfft, libusb, ...). +
-  * Some math skills, at least has to know what Fourier transform does. +
-  * Knowledge of most used data structures and algorithms, network and thread programming.+
  
-===== Test =====+Measuring Hardware - A set of sensors and hardware required for measuring.
  
-We have several [[en:rmds|Radio Meteor Detection Stations]] spreaded around central Europe (in range of French GRAVES radar)These stations  are GPS time synchonised with precision of 65 nsThe relevant signal will be reflected from meteor trails created by meteoroids which have speed in range of 20 to 70 km/sThis signal will be loaded by various levels of noise on each of station. The sampled signal will be used to on-line determining meteor trail vector in terestrial coordinates so the data bandwidth must by minimal required to this task. (This will be proceed in order to exrapolate impact elipse on the ground.+Hardware station guard - A hardware recovery unit which could separately reset every hardware part of measuring nodeThe station-supervisor daemon should use this device for error outputsConnection management between recovery unit and station-supervisor daemon will be self controlled by deviceInterface between hardware recovery unit and main station computer should be terminal based The terminal functionality should be provided by station-supervisor monitor software (ssmon) (if it is running, otherwise by 
 +standard system console) in system console mode station 
 +supervisor-should write debug output on this device (similarly to dmesg 
 +for example)
  
-How is optimal parameters of signal sampling ADC chained after [[en:sdrx|SDRX01B]] receiver for obtain optimal precision of trail geometry? (egbit depth of conversion and sampling rate)+== Network interface == 
 + 
 +We will not have direct IP connectivity from Internet to stations. Station supervisor software should generate data file outputs and upload these data files to an web server which serves station status information. Telemetry information will be displayed in the web on separate central web server.  Web page served by central status server is primary diagnostic interface. Terminal interface to measuring node is secondary backup interface which is not accessible on all stations. 
 + 
 +Moreover web access is intended as file based. File based means that 
 +station-supervisor uploads an diagnostic file to the central web server. Web server takes this files (from more than one station) and serve a web page with stations status information. User interface in this case is limited only to modification of an station-supervisor configuration file stored 
 +on web server.  
 + 
 +For example configuration file may contain an parameter for station reboot 
 +request. After station-supervisor will notice a change or update in the 
 +web server stored configuration file, station-supervisor reboots the station and 
 +update its parameters according to actual configuration file.  
 + 
 + 
 +=== Node software === 
 + 
 +Node hardware will run a data-processing utility ([[en:radio-observer]] for example), data uploading and distributing utility and station-supervisor daemon. Station-supervisor acts as watchman which checks proper functioning of whole measuring node. Data-upload is another daemon, which manages storage space on measuring node.  
 + 
 +Station-supervisor does not have a screen output. Screen output may be served by client program ssmon (Station Supervisor Monitor) which generates interactive status screen after connection to Station Supervisor server. 
 + 
 + 
 +==== Hardware implementation ==== 
 + 
 +=== Computing power === 
 + 
 +Measuring stations should be equipped with basic computing hardware architecture based on [[cs:arm|ARM]] specifically a multi-core ARM computer. The detection will take place at the stations already existing structures usually see [[en:programming_tasks# Considered data inputs (network nodes) |the list of  network nodes]] which will be connected to ARM computer. 
 + 
 + 
 +===  Network connectivity === 
 + 
 +Parameters of network connectivity depends on the specific measurements ongoing at the station. The preferred option but will connection to ethernet. Alternatively wifi connection to the local network and then to the Internet. In some cases, remote stations and measurement that are not required large data flows can also consider using [[cs:gsm|GSM network]]. 
 + 
 +===  Time synchronization ===  
 + 
 +Accuracy requirements for the station time again depends on the type of measurement. In the case of stations connected to the network via Ethernet, it is possible to use time synchronization based on NTP or PTP. Isolated stations with poor connectivity are then dependent on the use of GSNSS, as the time source. This issue is further elaborated on the page of [[cs:time_sync|time synchronization]]. 
 + 
 +====  Software implementation  ==== 
 + 
 +The network management would be advantageous to use the system to control robots [[http://www.ros.org/wiki/|ROS]], or any system designed for control [[http://www.astro.physik.uni-goettingen.de/~hessman/MONET/software.html|robotics telescopes]]. Similar functions are implemented in [[http://en.wikipedia.org/wiki/Simple_Network_Management_Protocol|SNMP]]. 
 + 
 +===  Data Processing  === 
 + 
 +Using [[http://root.cern.ch/drupal/|ROOT]] or [[http://ipython.org/|IPython]] 
 + 
 +===  Generating secondary outputs === 
 + 
 +The measured data of the stations should be distributed to networks specialized for processing certain types of data. (RMOB for example). List of currently available specialized networks follows.  
 + 
 +==  Meteorological data == 
 + 
 +In the case of meteorological data are available, for example the following networks: 
 + 
 +  * [[http://openweathermap.org/|OpenWeatherMap]] 
 +  * [[http://www.wunderground.com/|Weather Underground]] 
 + 
 + 
 +== Radio meteor observations == 
 + 
 +Data should be processed into colorgramm graph and forwarded to the network [[http://www.rmob.org/livedata/main.php|rmob]]. The current our version of software enabling the generation of appropriate outputs is or [[https://github.com/MLAB-project/rmob-export|Githubu]]. 
 + 
 +===  System stations administration  === 
 +  
 +Registration of individual stations and user management should be implemented into the currently existing web interface project [[http://www.astrozor.cz/|Astrozor]]. 
 + 
 +Each station would then authenticate into the system with its private key. Which would be generated either by the server based on the user registration to the system, or directly to a user who would put his public key available to the network.  
 + 
 +=== Measurement data storage and archive   === 
 + 
 +The measured data are primarily backed by the station (on the data node). Each station will send data in to decentralized repository which includes a central cache server - this server will at all time contains most of the data available (or can only provide a link between the data storage node with archival data and user requesting the data), the server has not guaranteed availability at every moment. The primary purpose of a central cache server is to provide data access to other users on the Internet (HTTP), who are not members of the data network but need to download them for further processing or viewing. 
 + 
 +{{ :cs:designs:measuring_network.png?direct&500 |Distributed measuring system network}} 
 + 
 +The first step towards decentralized data storage will be the implementation of a centralized server which will receive station data directly (FTP, SSH / Rsync)With the gradual development of software tools enabling decentralization of data, data nodes will be added to which the station will be able to write. 
 + 
 +Later the management of large volume of data will be handled exclusively in a distributed storage that will act also as part of the network. Data would therefore have redundancy based on division among multiple users, whose improve available data capacity and also data throughput. The data could then be exchanged between nodes by P2P technology, or other existing distributed data storage system. 
 + 
 +Decentralized organization such storage solution would be to split the data into the repository with its asymmetric cipher. Metadata about the content repository (including checksumswould be signed with the private key, which would be owned by its author. The public key or its hash, would serve to uniquely identify the repository.  
 + 
 + 
 +====  Data Visualization ==== 
 + 
 +=== Web Interface === 
 + 
 +Head echo meteor  marking in the web browser. It should be done by  clicking on the preview image generated by  the server from recorded data. (It should marked as straight line across of Doppler shifted reflection. Marking Doppler reflection is needed in order to measure the time shift.). 
 + 
 +Marking would be carried out by volunteers processors. Similarly, as in the project [[https://www.zooniverse.org/#space|SETILive]]. 
 +  
 + 
 +== Radio records replaying == 
 + 
 +Live waterfall and audio output generated in HTML5 webpage. (Similar solution is [[http://www.globaltuners.com/home.php|GlobalTuners]] or [[http://www.websdr.org/|WebSDR]]). This way of viewing would start after opening the RAW recording link of the detected radio signal. 
 + 
 +==  Display of meteorological data == 
 + 
 +Record of the wind direction should be combined with the speed and displayed in 3D, as the deformation of the cylindrical surfaces. 
 + 
 +A similar solution has a project [[https://nit.felk.cvut.cz/drupal/cs/moznostizobrazeni|display options time courses for visual analysis]] 
 + 
 +=== FITS rendering === 
 + 
 +Ideally, Fits viewer  would be a part of website and parameters could be set by scrolling bars in the browser. HTML5 would resolve data rendering. For example, setting the color palette, or composing of FITS to a film strip. 
 + 
 +Existing Web browser FITS: [[https://github.com/slowe/jsFITS|jsFITS]] 
 + 
 +The second possibility is to generate a preview by scripting on the server and erasing the oldest unused files (from the limited data space),  
 + 
 +===  Local Data Viewer === 
 + 
 +It should contain a computationally intensive functions that can not be handled a web browser.  
 + 
 +==  Viewing image based recordings  == 
 + 
 +Visualization system for multi-station experiments, where it would be possible to view the recorded data from multiple stations at once. Several windows with different data sets which could be deployed to screens of the viewer's workplace. Each browser window as separate, but it would have tied the slider, like tools designed for comparing files.  
 + 
 +Further improvement could be to use OpenGL for rendering textures of large images.   
 + 
 +== Current FITS viewers == 
 + 
 +  *  ds9 
 +  *  qfitsview 
 +  *  fits liberator 
 +  *  http://onekilopars.ec/qlfits/index.html 
 + 
 +== Multi parametric data viewers == 
 + 
 +A client application to download and special display data from measurement projects.  
 + 
 +====  Known projects with a similar goal ==== 
 + 
 +  * [[http://radioactiveathome.org/boinc/|Radioactive@Home]] 
 +  * [[http://qcn.stanford.edu/sensor/|About Quake-Catcher Network Sensor Monitoring]] 
 +  * [[https://nit.felk.cvut.cz/drupal/cs/univerzalniuloziste|Univerzální úložiště nejen pro lékařská data]] 
 + 
 +=====  Integrated SDR receiver  ===== 
 + 
 +The compact MLAB SDR receiver integrated in the box [[cs:unibox|UNIBOX01]] 
 + 
 +  * [[http://openhpsdr.org/wiki/index.php?title=Ghpsdr3|Ghpsdr3]] 
 + 
 + 
 +===== Astrozor ===== 
 + 
 +Adding support for users identification on observatories: 
 + 
 +  -The possibility of using an identification card 
 +  -Identify the user to the observatory on the basis of information about mobile phones IMEI, MAC IP, Bluetooth MAC (paired with bluetooth).  
 +  -A device mounted on observing site which would be capable to retrieve identification information. 
 + 
 +Other [[http://www.astrozor.cz/index.php?udalost=3|suggestions]] and [[http://www.astrozor.cz/index.php?udalost=2|bugs]] are recorded directly on [[http://www.astrozor.cz/index.php?misto=2|Astrozor]] 
 + 
 +=====  Meteograms for astronomers  ===== 
 + 
 +Generating view of clouds in the sky from a particular observation point. Meteogram in the form of video could also display the time evolution of cloud. 
 + 
 +For some cases,absolute concentration of moisture should be satisfactory. A movement or intensity of air masses of different densities, which then creates seeing. Computation should use NMM or WRF model. 
 + 
 +The result of this calculation could be used for observation planning.  
 + 
 +==== References ==== 
 + 
 +  * [[https://openmeteoforecast.org/wiki/Main_Page|Open Meteo Forecast]] 
 +  * [[http://flymet.meteopress.cz/|Aeronautical meteograms]] 
 +=====  Implementation resources ===== 
 + 
 +==== Investigators ==== 
 + 
 +The project is implemented by a team of several students CTU, TU and members of Robozor robotics club:  
 + 
 +  * Martin Povišer - Signal digitalization and storage decentralization 
 +  * [[https://usermap.cvut.cz/profile/milikjan/|Bc. Jan Milík]] - radio detection of meteors  
 +  * [[https://usermap.cvut.cz/profile/kakonjak/|Ing. Jakub Kákona]] - Organization and Architecture of the project  
 + 
 +==== Founding ==== 
 + 
 +The project was supported by the ESA program of ESA Summer of Code 2013 and 2014.  
 +Most hardware part of the project is realized from support of [[http://www.ust.cz/|Universal Scientific Technologies s.r.o.]] {{ :logo_ust.png?nolink|Universal Scientific Technologies s.r.o.}}
  
  
-Expected result is least one of: 
  
-  * Model of signal chain. (with noise injection) 
-  * Mathematical explanation of optimal values. 
-  * ADC parameters with explanation how was been determined.  
  
-===== Contact ===== 
  
-For futher information we strongly recomends to cantact project coordinator Jakub Kakona (kaklik@mlab.cz) 
  
-Pokud navíc umíte český jazyk, tak je vhodné si též prohlédnout [[cs:programming_tasks|českou verzi stránky]].   
en/programming_tasks.1375784051.txt.gz · Last modified: 2013/08/06 10:14 (external edit)