Controlling embedded hardware

1. Erlang in global radio astronomy Controlling embedded hardware Harro Verkouter/Joint Institute for VLBI in Europe

3. Joint Institute for VLBI in Europe Very Long Baseline Interferometry

4. 5m 100m Ever larger telescopes 15cm

7. Electromagnetic radiation

8. wavelength diameter pixelsize

9. A neat trick 100m

10. 20m  correlator 100m A neat trick

12. 10,000km

13. A different ball game!

14. Science

15. Science

16. Recording the data Record up to 4096 Mbps (512 Mbyte/s)

18. ~n2 Number of combinations to be computed as function of n

19. 16 telescopes at 1024 Mbps

20. 4 frontnode 4 backnode >20 Gbps/link 16x 10Gbps 16x 10Gbps 1Gbit/s ethernet UniBoard

21. Erlang?! • typical languages used • 80% C++ • 15% Python • PHP, Fortran, perl • database • MySQL • needed to convince management • pilot projects!

22. Decoding data files • VLBI has framed data format(s) • Look for ‘syncword’ • different flavours, same syncword, different framesize • Other format uses different syncword

24. (not quite) surprising little code • 212 lines of code • first dabblings ... • about a week’s work • recognizes + decodes 3 formats • decodes the BCD timestamps • supports distributed system • reading file on one machine • decoding on other machine • management duely impressed • got permission to do a larger pilot!

25. Soft real time monitoring website • webcams at stations • webpage with http links = #FAIL • unreliable links • really high latency (China, Chile) • don’t want broken images • low bandwidth • cache in Europe: multiple users eat all bandwidth • parse telescope logfiles • immediate antenna calibration info • generate timeseries plots as observation progresses • stored in MySQL

26. Soft real time monitoring website • webcam URLs + status stored in MySQL • supervisor starts webcam monitors • one gen_server for each webcam • loads info from MySQL • uses http:request/1 • strict checking on return value • only HTTP 200 status replies • content_length header set? • explode if problem • use a port to ImageMagick to scale image • write binary JPEG data into mnesia

27. Soft real time monitoring website • monitor all participating telescopes • telescope log file (ASCII text) • retrieved via python + ssh • read into Erlang using a port • low data rate after start up • parses lines • update MySQL • generate plots using gnuplot • for each telescope • query MySQL every so many minutes • using a port • write .png binary data to mnesia

28. Soft real time monitoring website • patch it all together using webtool • cache complete webpage in mnesia • too old? generate new one • serve .png and .jpg data from mnesia • image URL points back to webtool • extremely simple to cache + serve data • brilliant idea: mapping URL to erlang fn call! • in total used ~3000 lines of code • webcam, logfile monitoring; plotting; serving web pages • excellent test case for OTP and distributed system • got the green light for the real thing!

29. VHDL Hardware Nios2 CPU Nios2 CPU Nios2 CPU Nios2 CPU Software 1Gbit PHY 1Gbit PHY 1Gbit PHY 1Gbit PHY FPGA FPGA FPGA FPGA windows/unix UniBoard Control Computer EthernetSwitch EthernetSwitch

30. Polyphase Filterbank Nios2 CPU registers registers registers Delay Module registers 1Gbit PHY FPGA VHDL Hardware Software

31. Server VHDL Hardware Nios2 CPU Software UDP/IPv4 over ethernet Client windows/unix Control Computer

32. Simple binary protocol Read N 32bit words starting from START ADDRESS (Over)Write N 32bit words from the packet to START ADDRESS Read/Modify/Write N 32bit words from START ADDRESS + packet to START ADDRESS

34. FPGA client library one controller one fpga • Registermap: • list of register definitions • type of the register (details follow) • name of the register (symbolic) • address of the register in the memory map

35. FPGA client library Available registerdefinition types for the controller fpga:bit( <name>, <bit #>, <address> ) fpga:bitrange( <name>, <startbit>, <nbit>, <address> ) fpga:word( <name>, <address> ) fpga:range( <name>, <# words>, <address> )

36. FPGA client library Available register commands fpga:read( <name> ) fpga:write( <name>, <value> ) fpga:or_( <name>, <value> ) fpga:and_( <name>, <value> ) fpga:xor_( <name>, <value> )

37. Example -module(firfilter). %% this module is called firfilter, driving one of those -behaviour(personality). -export([registers/0, start_filter/2]). %% this module implements an FPGA personality %% Called automatically by the FPGA control framework registers() -> {ok, [ fpga:bitrange(num_tap, 5, 3, 16#24), fpga:bit(pps_enable, 28, 16#24), fpga:word(control_status, 16#abcc), ]. %% define a high-level command for this personality start_filter(FPGA, NumTap) -> %% disable the PPS fpga:execute(FPGA, fpga:write(pps_enable, 0)), %% read the control status register case fpga:execute(FPGA, fpga:read(control_status)) of 1 -> fpga:execute(FPGA, fpga:xor(control_status, 42)); _ -> ok end, %% however, you can easily execute a number of commands in one go: fpga:execute(FPGA, [fpga:write(num_tap, NumTap), fpga:write(pps_enable, 1)]).

38. Safety 2> fpga:execute(FPGA, fpga:write(pps_enable, 2)). OH NOES! error:"Value '2' does not fit in one bit (pps_enable)" FPGAControl terminating [{{badmatch, {reply, {error, {"Value '2' does not fit in one bit (pps_enable)", [{fpga,err,2, [{fil

39. Full correlator control system • besides single fpga control • parallel control of eight fpga’s • parsing + validating configuration files • using yecc • decoding + sending over UDP of binary data files • data as well as polynomial coefficients • capture network data to file • configure and control the hardware • MySQL read/write • two weeks VHDL coding = < 1 day Erlang coding

40. Full correlator control system data reader UniBoard data reader data writer UniBoard data reader command/control

41. The good • cross platform • we have Linux, Mac OSX and Solaris • distributed • typically many machines involved • non-variability of ‘variables’ • conciseness / amount of work per LOC • connects to MySQL • not easy to connect other languages to mnesia • hard crash on errors • about the most valuable property: can’t ignore! • binary pattern matching • every language should be jealous of this • re-use of development/testing code • it’s so easy to re-use code!

42. The bad • our VHDL engineers struggle • language simple, but • concepts like recursion + distribution are ‘hard’ • conciseness / amount of work per LOC • takes effort to read code written by someone who’s better at it than you • deploying a developing distributed system • we haven’t found a simple way yet • dynamic remote code loading could be awsome • it works for individual modules • useless if you have dynamic dependencies ... • erlang:apply/3

43. Thank you for your attention!