Echo Networks FailPoints – Remote monitoring and management (RMM)
The primary objectives are to create a near real-time, low impact method of:
- Ensuring that remote Internet/IP connectivity and communications are always up.
- Alerting admins as quickly as possible to potential network and/or security problems.
- Providing secure access to remote equipment rooms and networks without opening firewall ports.
- Providing backup DNS, simple remote camera views and even multiple environment sensors.
- All packed into a tiny inexpensive device that is on the job the moment it is connected. Nothing else to install.
FailPoints is a hybrid) RMM solution
FailPoints is a hybrid because our focus is monitoring connectivity, security and potential hacking and is not about watching server resources and updating machines. FailPoints can be used in addition to existing RMM tools or can be used on its own.
We developed FailPoints specifically because as an MSP, we wanted to see connectivity as it was being experienced by the locations we monitored. In most cases using remote monitoring tools and and counting on the network provider centralized tools would not show what we were seeing at the location.
Threats, problem alerts and assessments
In today’s connected world, it is reasonable that organizations could have dozens or thousands of devices installed in the field, some very remote and all depending on IP connectivity. When one or more devices fail to communicate, aside from possible data loss, someone needs to know about it as quickly as possible to evaluate the cause.
Failure to communicate could mean any number of things: from the monitoring device itself being rebooted or disconnected, something on the LAN, the network provider failing or even possible hacking.
The importance of alerts is critical, especially if one or more locations are no longer communicating or that unauthorized open ports have been detected. Even if most alerts can be attributed to something minor, it only takes one serious event to cause chaos.
Near real-time yet low resource usage
The solution starts by installing a small, inexpensive device called the agent at each location to be monitored. No need to install software packages on servers, the agent is just another LAN client, yet packed with everything needed. The agent’s primary task is to monitor Internet connectivity, logging outages as short as one second, while FailPoints monitors that the agent is still communicating, providing both a responsive and reactive solution.
FailPoints can be used standalone or alongside existing monitoring solutions for redundancy; logging problems which centralized services may not be seeing. FailPoints monitors from the location’s perspective, is inexpensive, easy to deploy and instantly on the job.
The agents reports help confirm or eliminate problems which could cause disruption of data flow, loss of productivity, cause security alerts and in some cases, even lawsuits, due to one or all of the above.
The bigger picture
Unlike central monitoring polling outward, agents are constantly communicating and each location is reporting what it is experiencing from its own perspective. An overview map visually shows any and all locations which might be experiencing problems making it easy to know if an entire street, a neighborhood or a section of city is affected by connectivity issues.
Part of the solution is to help correlate events in order to take appropriate actions. Very often, centralized monitoring cannot see what individual locations are actually experiencing. In some cases, some locations may all be using the same networks or providers and having the ability to correlate problems means it becomes easier to understand where, with whom and how many locations may have weak points.
The five main components of FailPoints
-Remote Access Service (RAS) offers secure, encrypted access to equipment rooms/LANs without opening firewall ports.
-Built in DNS server for resolve redundancy in case upstream servers become unavailable.
-Remote Router/Firewall unauthorized open ports monitoring and alerts.
-Internet connectivity monitoring with detailed outage data, trends and history from each locations perspective.
-Optionally enabled sensors and video camera for security and job evaluation before sending personnel.
More information about each feature
RAS – Remote Access Service
Secure, encrypted remote access to LAN devices without opening firewall ports.
Sometimes, opening a firewall port, even if access would only be allowed from one remote IP/network may not be feasible. However, each agent comes with a Remote Access Service (RAS) feature which the admin can enable from the dashboard. The admin enables RAS, sets the IP of the devlce they would like to reach on the LAN then specifies a specific public IP being allowed access.
The agent opens a secured, encrypted outgoing only connection to the FailPoints RAS services which in turn allows the admin to securely access the configured device on the LAN. A link will be shown in that agents dashboard when the RAS service is enabled and only that agents admin can access this function.
Example: Admin needs LAN access to a remote location to reach configuration servers and other equipment but opening ports which can be seen by hackers scanning is not an option. The FailPoints agent makes a secure connection to one location which in turn allows one public IP back into its local LAN.
Example: Hackers gain access to the local router/firewall and rules are changed preventing your admin from reaching the LAN. The FailPoints agent gives your admin a second chance to regain control and/or monitor the situation.
No matter the situation, admins have direct access to the LAN via secured encrypted access with no interesting ports to be found by someone scanning the location over the Internet. No open ports on the firewall means nothing interesting to see.
Built in DNS server
Each hardware agent comes with a pre-installed and enabled DNS server. The DNS server can be used for redundancy in case upstream DNS services become unavailable. Simply configure an additional DNS server for any devices on the LAN which you would like to have DNS redundancy for.
Remote Router/Firewall unauthorized open ports monitoring and alerts
Once you install an agent at a remote location, you can enable a security scan from its dashboard on your FailPoints control panel. Allowed ports are configured if any un-authorized ports appear during ongoing scans an alert will sent to the agent’s admin via SMS/Email. Each can be set to different recipients to ensure the right people are made aware.
Internet connectivity monitoring with detailed outage data, trends and history
Knowing as quickly as possible when one or more locations are no longer communicating is important, especially when it comes to critical services that must be available at all times. Locations could be customer sites, equipment rooms and/or closets, anything which is dependent on Internet/IP connectivity.
Centralized Internet monitoring does not show how connectivity is being experienced from each locations perspective. Using FailPoints, you not only know immediately if one or more locations are no longer able to communicate with the Internet but you also get actionable details.
Reports show exactly when, where and who is having an issue: in the building, with the provider or beyond the local provider no matter if outages have been reported or not. Reports show all events that each location have experienced along with trends and historical data which could help solve issues and even prevent finger pointing and lawsuits.
Enabling instant notifications can alert an admin about one or more locations no longer communicating.
Optionally enable sensors, image/video, motion, etc
Currently, a USB camera can be connected to each device in order to get a quick view of somewhere which is being remotely monitored.
However, additional functionality could be added. For example, it might be beneficial to take an image or a video clip of the surrounding area in order to help repair personnel to plan what they should bring before travelling to remote locations. Another example could be taking images or video when motion is detected in an equipment room or closet where no movement should ever occur unless authorize personnel are on site. Images and video could immediately be sent to the dashboard for review.
An optional environment sensors feature can be included with each agent for a one time cost. Temperature, humidity, pressure, light and others as shown on the right are sent to the dashboard for that respective agent.
With a bit of additional development work, practically anything could be added depending on the requirement and budget.
The FailPoints service is for organizations that must ensure constant connectivity, security and accountability.
Accountability for communities, cities, states
Searching the Internet yields countless cases where outages had serious consequences which lead to finger pointing, arguing and sometimes even lawsuits. Organizations, cities and states have ended up heading to the courts over disputes with Internet network providers. Lawsuits can be expensive, time-consuming and In most cases, there isn’t enough proof about what happened to comer to any meaningful conclusion.
FailPoints helps to eliminate these and other problems for businesses and consumers who depend on network owners being reliable, Reports show a clear picture from multiple points making it practically impossible to argue or dismiss problems even after they have occurred. This helps everyone involved to move forward, finding and fixing problems.
Broadband initiatives, ISP, MSP, IT
FailPoints helps Internet broadband initiatives by providing important information that network operators can use to maintain and refine their systems. Our solution can be especially useful for network operators that must ensure their services are reliable in order to gain consumer confidence and motivate people to use their services.
For IT and network operators, FailPoints is an affordable method of knowing exactly where problems might be, from the customers perspective. You’ll know if the problems are at the customer location, with the provider network or even beyond, something that centralized monitoring is unable to do in a near real time way without using up a lot of resources.
When Internet issues arise, especially VoIP related, customers using managed services can quickly point the finger at their MSP. You need a speedy way of determining if the problems are at the customer premises or with their Internet connection. By knowing where the problems are and taking care of them quickly, your customer will always trust that you are on top of things as you should be.
Inexpensive, easily deployed
Unlike PCs running software, hardware agents don’t get accidentally turned off, missing important events. They also automatically update, can be remotely rebooted and run 24/7 without any human interaction using only about 1 watt. Perfect for remote, unmanned locations or locations which have a limited amount of power such as solar.
Reports show when, where and with whom problems are occurring, along with simple to understand text such as “the problem is at your location” or “the problem is with your Internet provider” or “the problem is beyond your Internet provider.” Details include which hops might be experiencing problems along with other optionally enabled functions such as fully automated speed testing.
Hardware agents are a one time cost and each agents reports are billed at a low monthly cost. This is a cost which can easily be absorbed by saving time and money or passed on to the customer as an MSP.
While one second outages might not sound important, consider a connection which is experiencing constant and or intermittent micro outages and how those could add up to possible problems with VoIP and other real time protocols. We’ve seen it and experienced it many times.
Should a device fail, it is as simple as ordering a replacement. For redundancy, to ensure that changing out the device doesn’t lead to starting all new reports for a specific location, a second device can be ordered and keyed to the same ID and kept locally as an instant replacement.
The overview can be reached using PCs or mobiles by anyone authorized to use the service.
Organization may have dozens or maybe even thousands of locations being monitored. For this reason, while it is possible to view all locations, the view is set to show only locations which are experiencing potential issues so that someone monitoring can quickly spot trouble.
The overview can be used by both technical and non technical personnel to keep an eye on operations.
The overview map becomes especially useful when problems occur that affect more than one area. It clearly shows if one street or one block, section of city and so on are being affected.
Because locations are likely widely dispersed, each agent can be configured with its own admin contact for SMS/Email alerts for efficient dispatching of truck rolls. Alerts must be dealt with and a comment entered before it can be removed as an alert. This is to help management and other admins to know what the history of each event was and what was done to remedy.
Agents are fully self contained
The FailPoints agents are inexpensive, small embedded devices that don’t get accidentally turned off like servers and PCs might. Unlike servers and PCs, they are purpose built meaning no potential software conflicts such as mixed software running on a server/PC. They work 24/7 never missing important events, automatically update, can be remotely rebooted and use as little as 1 watt. They are especially useful at remote, unmanned locations that have limited amounts of power such as solar and/or no PC/Servers to install software onto.
No large budgets needed
Agents come pre-configured, pre-loaded and are fully self contained with everything needed. There are no servers or software of any sort that need to be installed in your office or at remote sites. Other than the one time cost of the hardware, there is a monthly fee of $7/month or $60/yr for each agents reports.
The service includes perpetual bug fixes, updates, new features and email/phone support during business hours. Our developers are constantly on the job and our infrastructure is well maintained. We very much welcome customer feedback in order to continue to improve our services. If you wish for something, we may be able to add it as a custom feature just for you or it could become a feature that all customers can use.
Managing large numbers of agents
The overview map gives a quick visual on how various locations are doing. When there are large numbers of agents/locations being monitored, a long list of agents as seen on the right would be more difficult to manage.
Aside from being able to reach any agent using the overview page, an agents list is also provided. The agents list shows the status of each agent/location and gives admins access to any agents dashboard. The list can be sorted by several fields to make displaying a very large list more manageable.
Admins can clearly see any and all agents and managed locations which may be experiencing problems and needing attention. Checking those agents dashboards will give more information.
Our intention is to make dealing with small or large numbers of assets as easy as possible while at the same time showing a bigger picture of problems which may be occurring. With feedback, we can better refine these and other functions.
The dashboard shows details related to each agent and its location. Reports show when, how long, where and with whom problems are occurring, along with easy to understand text such as “the problem is on the LAN” or “the problem is with the Internet provider” or “the problem is beyond your Internet provider.” Details include IP, hop, hop count, location, owner and more.
Find problems fast
Once a monitoring agent is installed at a remote location, it will begin generating performance data from that locations perspective. While large outages can be found on outage sites, each agent will report from its own connections viewpoint and perspective.
Problems become more obvious, the ability to correlate means being able to quickly confirm or eliminate problems that could otherwise take long periods of time to assess, especially after the fact.
As a VoIP and managed services provider ourselves, we needed a way to let customers know that their issues were most often with their own Internet connection and not something we supported at their location.
Many companies provide a variety of unified communications and the importance of making sure everything is running perfectly cannot be understated. Finding and knowing where potential problems might lurk is just as important as fixing them. Determine if problems are onsite or if an network provider must be contacted. Preempt issues, spot possible failures in the works and keep communications working 24/7.
Stay ahead of potential problems
The FailPointssolution can be used on its own or in addition to centralized monitoring and helps ensure that Internet connectivity at one or thousands of remote locations is always up, always available. This is especially important in situations where two or more circuits are in use for redundancy. If redundancy were to fail, it could lead to serious problems. FailPoints helps to eliminate these and other issues.
FailPoints gives IT personnel a heads up of possible issues, always with hard data during and after problems, often even preempting customer complaints and down time.
- Overview map shows live status of one or any number of agents installed in the field.
- Map shows a clear visual of problems which are affecting multiple locations.
- Instant email/SMS alerts of unauthorized open firewall ports, connectivity problems and others..
- Secure access without opening firewall ports to remote equipment rooms/networks to access config servers or regain control in a hacking situation.
- Monthly 7G, Weekly 1.6G, Daily 462M.
- Send average is around 6 kilobits/s, receive average is around 16 kilobits/s.
- Connectivity checking is 24/7 to log outages as short as one second,
- Network polling (send/receive) is around every 15-20 seconds.
The above figures are from a device that is monitoring Internet connectivity and sending sensors data at the default rate.
In most cases, this is acceptable (especially when trying to find problems) but if there is a custom requirement, we could slow or even speed up the polling. If there is very little bandwidth we could also design a custom solution which is tuned to use as little bandwidth as possible.
Quick and easy deployment:
- Installing an agent takes only minutes and begins working instantly. Connect to free DHCP port, Activate.
- IP connectivity is monitored 24/7. All 1+ second outages are logged.
- Optional automatic speed testing for baseline speeds and based on conditions such as latency.
- Agent is usually able to determine if problems are local, with the provider or beyond.
- Reports show trends such as when most outages occur, how long, where and with whom.
- Reports show exactly where problems were/are, length, IP, hop, owner and other details.
- Dashboard can be reached over the Internet using mobile if local network is down.
- Each agent can have its own unique admin email/SMS alerts so that multiple people can be notified.
- Alerts must be dealt along with comments for admin and management for historical purposes.
Additional hardware benefits:
- The FailPoints hardware device is on the job 24/7 for the best logging possible.
- Auto starts, auto updates. Only 2″-3″ square, uses around 1watt, perfect for remote, unmanned locations.
- The agent software runs in memory limiting flash erosion in order to survive for years.
- No learning curve, no servers or software to install, everything is self contained on the agent itself.
- Agents start at only $49.95, (discounts available) are a one time cost per location to be monitored.
- Reports are only $7/month or $60/yr and include updates, ongoing development and support.
- Agents are easily replaceable with no loss of historical data.
The overview map gives a visual representation of agents and conditions. Because some organizations could have thousands of agents installed, only agents which are experiencing problems are displayed making it easy to spot problems very quickly.
For example, in the image below, we will see a string of yellow markers indicating an Internet problem affecting multiple locations.
The monitoring person who does not have to be technical can zoom in to see more details then click on any of the markers to find the agent ID and other details. Different colors represent different alerts such as a location is no longer communicating or an unauthorized firewall port was found open for example.
When one or more agent shows on the map, it also means that alerts have been sent out to the person in charge of this location along with a copy to a secondcary person such as management so that someone can respond quickly.
In addition to the overview map, the agents list also shows the status of each agent making it very easy to spot problem locations. Filtering is also available by agent IDs, postal code, city, streets, lan/lon and so on.
Remote access service (RAS)
Where ever you have an agent installed, you have the posibility of gaining access to the LAN or any peice of equipment at this location. RAS takes only moments to set up by configuring the IP of the device to be reached on the LAN, the port and which public IP is allowed to access.
It takes a few minutes for RAS to become active and once it is ready, the link will show in the dashboard. Simply click on the link and it opens a connection to the remote device without requiring any ports to be set on the local firewall. This link could even be used by an admin to gain access to a network which was accidentally lost through a bad firewall configuration or even a hack in progress.
The LAN IP can be changed at any time in order to reach a different device and a list of previously used entries is also shown for quick access.
The admin can enable a security scan, setting the allowed ports and receiving alerts if any unauthorized ports suddenly become open on the local firewall. The scans are performed on a regular basis, set by the admin and alerts are sent within a minute of unauthorized ports being found open.
When alerts are first received, they are shown in red indicating that it has not been acknowledget yet.
Acknowledged alerts change to yelow indicating that someone has seen it and is working on the problem.
In the historical section, all alerts ever sent for this location will be listed.
When outages occur, all details are shown by mousing over each event. Details show hop, IP, who owns the network and the algorythm tries to determine if the problem was on the LAN, with the provider or beyond. Average times of problems are also shown below and are built up over time.
Over time, it becomes obvious which routes are the most problematic, most often down and who owns them as well. This can help when making upgrades or even when deciding on future services with third party companies.
While the dashboard shows only the last 50 events and other limited details, the historical menu shows all history gathered since this agent began monitoring. The viewer can switch between views such as one week, two weeks, a month, six months, a year, etc.
In certain situations, it could be good to have eyes on a remote location. Being able to take a quick look using a video camera could help technicians to see the extent of damages and what tools and materials to bring or it could help identify other things which could be useful before rolling a truck to the location.
Adding a camera is as simple as connecting a UVC based webcam to the USB port when installing the agent or later then enabling the service as needed from the dashboard. If there is a need to monitor multiple cameras, views could be combined and all locations could be monitores simultaneously.
Optional sensors can be added in order to monitor environment conditions for every location that has a FailPoints agent installed. Sensors include a number of useful metrics which could help technicians.
At a glance
The At-a-glance section of the dashboard shows a quick view of location, network, DNS servers and other settings for this agent. If the public IP is dynamic, it will be updated in the dashboard automatically and in DDNS settings if the option was enabled.