Daniel Schroder

Was taken on at vanilla internet service provider as roll of Senior Network Engineer when permanent, during the probation period I primary looked over whole network, with the idea of admin, learnt the whole network infrastructure, and did support on ubnt/ubiquity devices and controllers at clients. Due to shortage of ubnt/ubiquity hardware was decided on tplink and I setup and tested tplink wireless devices and controllers for network wide roll-out, and documented it. This is all devices over the wan network being setup and run from a controller, with all the needed configs to make the work of the installers as simple as possible, this was done. The technology or framework to do all this is called tplinks omada EAP series centralized management.

References:

• Alan Levin Founder/CEO
• Diren Naaidoo, Network manager.

Was initially contracted to build a single network for Tableview high school, but then afterwards was further contracted to roll out that network as a template for numerous other schools using mikrotik hardware. This includes SIP/VoIP, bandwidth management, security and monitoring, Active directory policy administration and implementations for network specific tasks. Also setup and deployed VoIP networks, took over administration of the larger unifi networks, access points and controllers.
Started (and completed) moving as many schools as possible to what I saw as industry best practice for large complicated networks. This included vlanning off various critical parts of network onto their own networks and dhcp segments, routing all these via routers and switches. Isolated critical parts of the larger networks. Subnetting or vlanning components, printers, voip devices, cameras, this was either for stability ( telephony ), or security ( cameras ). Also, as far as possible vlanned and separated wifi networks and enforced SSID attached to vlans for security. Students/Staff/Wifi/Cameras all separate.
Simplified bandwidth management by marking traffic into bulk and low latency (VoIP, DNS etc). Most networks have two uplinks, 1G and 100megs, running in failover, and UPS. This ensures that a 1G uplink can easily operate on a busy Lan, or an office can operate comfortably with both incoming vpns, VoIP, office 365 and normal internet access. Started rolling out IPv6 on Wi-Fi networks, tested on school Lans, surprisingly very few issues, single one was happy eyeballs for some browsers which was mainly due to staff not understanding Ipv6. IpV6 is going to become standard by mid-22, and made sure infrastructure configurations can roll it out. Standardizing on Ipv6 for WiFi SSIDs, since WiFi is 90% android, which does not care if it’s v4 or v6. On windows part of networks used active directory network policies to mark traffic according to groups based on network access allowed. This allowed routers and access points to prioritize traffic, or for routing. Groups I use were staff/machine, or staff/student/machine and this only affected internet net access speeds and prioritization as well as routing. Windows machine marked traffic based on user or machine access (like updates), router used this to create groups, assign to priority ques, or mark routing if uplink lines are assigned to individual groups. Also allowed me to quickly group network device ip addresses into groups joined to a domain as authenticated or not, as well as internet access for actual users, or just software running on machines, and manage them as groups, not hundreds of firewall rules, and assign non domains users to lowest priority. Security was the difficult part, lot of scripting and automation to ensure no easy means to access all the networks from outside. This was mainly done by configuring all remote networks into one big network via a vpn, with syslog used to send all network events to a centralised server running a virtual system that automatically separates each stream into it’s own location, and creates a file for each day, going back a year. This is more than +20 routers, +100 switches, +100 access points, +200 VoIP devices and a few Pabxs. Once a day spot checks were done by sifting through logs and looking for the usual suspects. Known issues were scripted and run continuously to raise issues, once problem I came across is efficient automated notifications to all parties, most problems are already automatically raised but not acted on until a user complains. Incoming syslog from data all networks is about 10mb/s. Network wide DNS was routed to Cloudflares family DNS (student wifi and labs) or Malware public servers, to catch any compromised devices or machines for the known attempts to phone home once plugged in. This is not a perfect solution, but does mitigate risk before virus software can kick in. Were possible depending on router capabilities (CPUs) layer 7 was used to catch SIP probes as an example, more than 4 4xx code replies a minute was blocked. Torrents on wifi was also blocked based on layer 7, since torrents try any means possible to bypass standard blocks. As much as possible when users were firewalled, only there internet access was cut off, this is ensure they can at least function on local network while they wait for support. Where 1g+ internet access links for more than a few hundred users, optimized firewalls to ensure load did not effect latency. Setup vpn (pptp/l2pt/IPsec ) access on all routers, and networks, for staff at home, as well as networking the whole network I administrate. The complete network was a big vpn I setup to make my life easier for administrations. It was done using OSPF over all possible routes between schools. This insured as long as two possible paths, no matter over what lines, connectivity to that network was still able. This also functions as a fully functional backup route for the whole network if primary internet link is down. This also mapped all machines over all remote networks to ip address accessible from the “admin” network where I was based, or at home, this made monitoring as well as access to switches, access points, remote desktops or any other network access to any device possible over the entire live network. This was also used for staff access from home, using radius or user/pass to there part of the network. Important to note, all networks were based on the same network ranges, but remote network blocks are remapped to each network, this made it possible to have access to the whole network with no disruption to local network configurations which made support easier for staff. Started rolling out NAS hardware specifically to make it possible to either run specialized disk access on busy servers possible (iSCSI), and to/or run controller/dns/critical software and services on the server so that there no disruptions for when servers are rebooted. A proper setup NAS server can have half of it fail and still function. All unfi networks, checked once a week for problem areas and did what was needed to get wifi access acceptable. This involved disabling auto network optimization when networks to big, looking for strong overlap areas and reducing, or increasing levels if brown spots become obvious, disabled legacy support were not needed, and optimizing RSSI levels accepted or rejected to improve roaming to the point VoIP is capable. Standard admin on all wifi networks. Some networks are:

• Tableview High school. 1200+/- network devices. 200+/- windows machines 400-500 wifi users, students, staff, guests vlanned and on there own SSIDs, 13 Access points, 23 vlanned VoIP phones, 11 Switch SFP+ (10G) backbone. Two vlanned 30 machine labs. Core router is a 39 core mikrotik.
• Parow High High School. 300+ devices, 123 windows machines, 80 wifi users, 6 ubiquity switches, one mikrotik router.
• Swartland High School. No wifi network, one PABX with a sip trunk, 80 Machines, two 30 machine labs.
• Dfmalan High School. 1300 network devices. 300+ Windows machines, three 30 machine labs, +700 wifi users, on three SSIDs, each on there own vlan and DHCP server. ( Staff/Students/Guests ). Students and guests no local LAN access, only Internet. Each fibered building on it’s own vlanned segment. 14 Ubiquiti edge switches, 23 access points, +25 VoIP phones that on there own vlanned network. Other schools, as far as possible setup in the same method.
• Parow west school
• Parl boys primary
• Swartland low school
• De Kuilen high and primary school.
• Fairbairn college.
• Innofin financial. Few others

Started branching into general systems administration for clients that varied from moving mail domains and exporting and importing mail to different providers, general Laravel/WordPress or other hosting admin, and building DC hosted linux servers for docker or just simple hosting. As well as SIP dialplans admin design and troubleshooting. Enjoy this work, exploring options.some blue text.

• At Odm develop and deploy software on large media screens throughout South Africa,customers include wimpy/spur/kfs/cape union mart etc, and the screens are maintained remotely by software developed at ODM, it was my task to simplify installations, and and facilitate the transitions from legacy linux systems to a newer standardized UBUNTU, which I did, by packaging the software into standard debian, and streamlining remote upgrades of packages so new realse can be rolled out over +8000 screens (wifi or 3g), a sensitive process, since a mistake can cost thousands in data.
  
• Developed and built a automated UBUNTU installation server that remotely booted new units, formatted drive, placed a ubuntu installation on the new systems, installed the odm software, updated the system, and shut the new unit down ready for shipping, 15 Installations took around 15 minutes from boot till unit shutdown, no human intervention needed at all, prior installation staff had to boot off thumb drive, and do each unit manually, reducing a half an hour installation to 4 minutes (one unit), saving 20+ minutes.
  
  References
  • Ross Simpson development team leader
  • Charles Parsons HR head.

• Setup system wide monitoring, zenoss .. for linux and a wireless network, snmp and traps, made sure issues were resolved and wrote reports for management as to solution and causes.
• Dealt with all major escalations and made sure (as possible) repetitions not re-occur.
• Administered a wireless network, which included new sights, link optimization, stabilizing and where possible implement best practices, this included OSPF and MPLS for routing.
• Administered a ADSL network, bind, radius, and billing integration, also involved with billing issues that usually only I could resolve.
• Administered and optimized mail servers (15k+ active users) and monitored failover issues, and implemented performance changes for peak times.
• Was on call for all final escalations.
  
  References
  • Sean de Gruchy – director
  • Tim Huppert – support manager

• Worked under the technical manager, maintained and monitored systems used. Centos servers, Xen servers, cisco routers, HP switches, wifi devices and cloud xen servers.
• Managed and monitored all the unix systems, system tuning based on load, real time adjusting of resources, raising system issues (all devices) , writing in monitoring were necessary, writing scripts to monitor things like mailques, link stability and to post results to monitoring software like prtg, zenoss.
• Actively pre-empted issues, load, link instability, manage alerts, raise issues to the correct staff where necessary, advanced system monitoring, like cache efficiency, hosting server load efficiency, long term graphing of any load, protocol or speed.
• Implemented ISP wide SNMP configs, as well as trap management, and managed rerouting of notices, reducing spurious notices, and rewriting mibs
• Managed and monitored device backups, went in after new devices and configured anything that was not done, things like snmp, syslog , ntp and so forth.
• Daily network wide checks, sifted through reports and notices for anything serious, wrote systems to monitor and graph bandwidth, adjusted thresholds (which raised notices)
• Forensic reports on system failures, breaches or line failures, or any issue that causes an error somewhere and source of error to locate, extensive use of SNMP, syslog and SNMP traps to establish timelines of events.
• Fixed any issues on any Unix system ISP wide, and either fixed or located problems on switches routers and wan links, investigated issues on wireless links, channels, error rates.
• Did weekly scans of network and added any devices found to monitoring, set notice levels, went and checked device configuration, and made adjustments or raised changes that had to be made onsite to staff.
• Worked extensively on bandwidth issues, unix system tcp/ip stack adjusting, troubleshooting.
• Setup different SMP systems and adjusted for load, checked, adjusted and setup unix systems to use resources as efficiently as possible, and systems under heavy load, adjusted for higher efficiency, or removed source of high load, be it swap, non-tuned parameters like connections per second, memory usage and set all relevant graphs and thresholds , for wccp proxie systems, mail systems, web hosting servers.
• Picked up incorrectly setup unix systems, either fixed or referred back to staff, or setup templates for network wide implementation.
• Any general issue that affected any part of the network.
  
  References
  • Jonothan Maliepaard – founder
  • Jaap Schultan – tech director ( Currently Datacentix )
  • Saul Stein – Technical manager

• Worked closely with the Technical director, and was instrumental in its redesign, roll-out of Wi-Fi WANs, and data-centre devices and servers, as well as built the name/mail servers and implemented the monitoring of the complete network, and designed the wan using multilink, wrote firewalls with full backup routines as well as designed the firewalls for gateway devices, as well as the day to day running of the complete network, systems and links.
• Maintained the Debian systems, located in JHB, and CPT Datacentres these were for bind (DNS), webservers, monitoring servers sip and radius. Also built and maintained the monitoring systems that monitored, escalated and notified mangers on the national networks. (Zenoss on Debian)
• Built and maintained the unix gateway systems that the ISP users accessed the internet with, which consisted of firewalls, proxy servers and multilinks for inter branch communications, as well as sip (VoIP) traffic links, media servers running on Debian.
• Closely involved with the Wi-Fi wan roll-outs, Mikrotik routers, setups, maintenance, and automation as well as scripting.
• Also built and ran the Debian systems that asterisk client servers as well as wrote the dialplans for VoIP links.
• Built the mail servers, which were based on postfix, database backed, with virus, spam scanners on separate systems, as well as the monitoring and daily reports of errors, based on postfix.
• Wrote all the scripts and systems (perl) for maintaining links (ADSL pools, and backup ISP links) with dynamic load balancing and load failover, as well as line failure detection and rerouting.
• Setting up of new remote sites, routing, upgrading of mikrotik, entering into the monitoring, routing and setups of public AP’s in the beginning, this has now been automated.
• Liaised with departments and managers on correcting and managing line faults, power failure notification, voltage, signal and temp faults, and made sure they were corrected, and raising issues before they become problematic.
• Maintained all the Linux systems default firewall set tings, and made changes were necessary.

• Responsible for ICSs critical services infrastructure, as well as ran the firewalls.
• Also rebuilt all the Unix servers, was closely involved with the Linux systems in a problem solving capacity, managed bandwidth, managed network access, managed the proxy systems, wrote extensive ACLs, and extensive server tuning, also implemented failover CARP based virtual systems, and did a lot of virtual NETAPP backed system implemented. Serviced 30k students.
• Was closely involved with developed and rolled systems and software built in-house, as well as worked closely with the then DBA on the platforms. (Primarily Oracle on Suse)
• Looked after the mail spools for the University of the Western Cape, this was qmail with custom, written plug-ins. These were initially Linux based, but were converted to FreeBSD virtual systems to cope with load (by me)
• Looked after internal and external DNS services, on both Suse and FreeBSD systems, and made all required system changes were needed, new requests and DNS changes.
• Looked after IPtable, and PF firewalls, as well as designed, maintained and looked after bandwidth requests from campus, the firewalls were also built with FWbuilder to ease departmental management overview, this included the Wi-Fi network (300 plus applications, the DMZ, links to UCT, pentech and affiliated institutions.
• Looked after, built and ran Linux and FreeBSD proxy servers throughput campus, this included advanced ACL, hot failovers, and load related maintenance.
• Designed and built all critical services hot fail over systems using CARP, Virtual systems booting off SAN backed file systems.
• Deployed and hardened new systems to go into the public DMZ, Linux (Ubuntu/Suse/Debian/FreeBSD/Solaris) as well as wrote and maintained all firewall considerations, as well as looked after the general security around the DMZ and firewalls.
• Worked closely with the developers in deploying live systems on Suse/Ubuntu and Solaris. (E-Learning, student systems, staff systems, and library databases.)
• Took care of all departmental special requests, also maintained and took care of the library special needs. (Web database access, Linux ezproxy servers, changes needed)
• Assisted with general network issues, as well as assisted on core network general admin, emergency routing, and failure correction, reporting and monitoring.
• Worked closely with DBA’s, as well as backup for emergency changes, mainly MySQL/Oracle and Postgress on Suse/Solaris Ubuntu servers.
• Generally maintained all the UNIX systems when time allowed, (About 150 in total) as well as the e-learning Solaris clusters.
  

Reference

• Istvan Berko

• Built and ran two labs, one for NASSP, one for the Maths faculty, unix thin client based. Was responsible for the general maintenance, as well worked with lecturers, maintained math routines, did a lot of porting of software, graphical software that students used for astrophysics, built and ran the internet/mail systems, and worked with remote departments at other universities to facilitate visitors and exchange student assistance, as well as looked after the dept general Unix needs. Had to work with UCTs ICTS, lots of fights (Hi Andrew Alston back then :) friends now)
• Did a lot of high performance work, complex calculations, and lectures took place on the lab networks, as well as built and ran as well as tested a few cluster technologies for complex astrophysical calculations.
• The labs where two server failover with 30 points for server, with one able to carry 60 in event of failover. FreeBSD XServer based with Linux remote boot thin clients.
• Managed two labs of 30 workstations each. Workstations were customised Linux systems (customized for graphical speeds, and cluster computing) working off FreeBSD X based servers.
• Assisted students with (astrophysics) projects, this included general programming assistance, as well as porting lecturer software to local systems, running astrophysical based Linux apps off FreeBSD for load gains, porting C to IFC compilers for speed gains.
• Supported specialized maths apps (Matlab, Octave etc), also supported different word processes and plotting software (open office/gnuplot etc)
• General Unix support for the Maths dept, as well as ran the math Post Grad labs, ran internet access systems, printing maintenance and administration,
• Wrote and maintained the webservers for student needs, student project hosting, online labs status pages, free seats and general info needed remotely, and schedules.
• Liaised with the universities ICTS dept for student access issues.
• Liaised with the Observatory and SALT for visitors, network needs of travelling students.
• Deployed various clustering software for big computations that were needed, as well as massive image processing.
  
  References
  • Prof Peter Dunsby

• Maitland trust is/was the South African branch of the Maitland group, which is a private and corporate Client Services business
• Built the Server/client windows policy administrated active dir network throughout South Africa, three branches, with mail systems, firewalls, and wan links plus failover for links, with interbranch document sharing (over 60k plus documents) and very tight security (Active dir forced policies on all workstations), and firewalls for all the branches.
• Mail systems were in house, and again, FreeBSD based. The WANS where multilinked over IPv6, redundancy done via RipNg, with two way failover for each branch, to ensure all documents were readily available to staff that travelled or needed to work on remote documents.
• Built all the Unix servers for mail, security and wan links, deployed at all the South African branches, as well as looked after all the branches from Cape Town (70+ users throughout South Africa), was primary involved with looking after security, and monitoring.
• Built and administrated all the windows servers at all the branches, with WAN failovers.

Was involved with building and general maintenance of IT related tasks with the Investment companies, also ran a few personal projects from there (unix.za.net)

At for Dockside internet, as network admin, as well as general ISP related work, support and trouble shooting, this included client setups, routing (bgp table maintenance), firewall maintenance, and general ISP system admin duties. (DNS/mail hosting, dial in banks, setup, and general ISP security, left after they were bought out by Datapro I think it was. Based on the waterfront and also looked after the waterfront wan (client access, faults and general maintenance.)
At Internet Africa (UUNET Internet Africa), did UNIX Support for about two years, back when ISPs where mainly unix based.

After school refused military service, was then sentenced in high court as a conscientious objector to three years community service and was sent to Sea Fisheries were I become involved with different aspects of the work done, namely building and deploying current metres while at sea and in workshops, and also took over running the satellite processing and receiving equipment for a Doctor Agenbach and was kept on after South Africa freed Mandela and abolished compulsory military service.
Sea Fisheries (3 years), where I worked on the Remote Sensing systems, was involved with Satellite data capture, processing, wrote software for graphical image manipulation. (The focus was infrared NOAA satellite data, I was assistant to a Dr Agenbach, and did all the low resolution infrared capturing, setting up, planning satellite capture passes, as well as liaised with ships as to where Ice cracks where on the South Pole, as well as Fishing vessels who wanted to know where the warm and cold water front for fishing. Also went to sea to deploy current meters, as well as assisted with general deep sea research. This started me off in IT directory after school were I passed matric. Most of this work was to do research on climate changes (el nino effect) and for generating and managing red tide outbreaks on the lower west coast in South Africa.