THIS POST IS CONTINUED FROM PART 16, BELOW--
CAPT AJIT VADAKAYIL SAYS AI MUST MEAN “INTELLIGENCE AUGUMENTATION “ IN FUTURE ..
Let this be IA
Let this be IA
OBJECTIVE AI CANNOT HAVE A VISION,
IT CANNOT PRIORITIZE,
IT CANT GLEAN CONTEXT,
IT CANT TELL THE MORAL OF A STORY ,
IT CANT RECOGNIZE A JOKE, OR BE A JUDGE IN A JOKE CONTEST
IT CANT DRIVE CHANGE,
IT CANNOT INNOVATE,
IT CANNOT DO ROOT CAUSE ANALYSIS ,
IT CANNOT MULTI-TASK,
IT CANNOT DETECT SARCASM,
IT CANNOT DO DYNAMIC RISK ASSESSMENT ,
IT IS UNABLE TO REFINE OWN KNOWLEDGE TO WISDOM,
IT IS BLIND TO SUBJECTIVITY,
IT CANNOT EVALUATE POTENTIAL,
IT CANNOT SELF IMPROVE WITH EXPERIENCE,
IT CANNOT UNLEARN
IT IS PRONE TO CATASTROPHIC FORGETTING
IT DOES NOT UNDERSTAND BASICS OF CAUSE AND EFFECT,
IT CANNOT JUDGE SUBJECTIVELY TO VETO/ ABORT,
IT CANNOT FOSTER TEAMWORK DUE TO RESTRICTED SCOPE,
IT CANNOT MENTOR,
IT CANNOT BE CREATIVE,
IT CANNOT THINK FOR ITSELF,
IT CANNOT TEACH OR ANSWER STUDENTs QUESTIONS,
IT CANNOT PATENT AN INVENTION,
IT CANNOT SEE THE BIG PICTURE ,
IT CANNOT FIGURE OUT WHAT IS MORALLY WRONG,
IT CANNOT PROVIDE NATURAL JUSTICE,
IT CANNOT FORMULATE LAWS
IT CANNOT FIGURE OUT WHAT GOES AGAINST HUMAN DIGNITY
IT CAN BE FOOLED EASILY USING DECOYS WHICH CANT FOOL A CHILD,
IT CANNOT BE A SELF STARTER,
IT CANNOT UNDERSTAND APT TIMING,
IT CANNOT FEEL
IT CANNOT GET INSPIRED
IT CANNOT USE PAIN AS FEEDBACK,
IT CANNOT GET EXCITED BY ANYTHING
IT HAS NO SPONTANEITY TO MAKE THE BEST OUT OF SITUATION
IT CAN BE CONFOUNDED BY NEW SITUATIONS
IT CANNOT FIGURE OUT GREY AREAS,
IT CANNOT GLEAN WORTH OR VALUE
IT CANNOT UNDERSTAND TEAMWORK DYNAMICS
IT HAS NO INTENTION
IT HAS NO INTUITION,
IT HAS NO FREE WILL
IT HAS NO DESIRE
IT CANNOT SET A GOAL
IT CANNOT BE SUBJECTED TO THE LAWS OF KARMA
ON THE CONTRARY IT CAN SPAWN FOUL AND RUTHLESS GLOBAL FRAUD ( CLIMATE CHANGE DUE TO CO2 ) WITH DELIBERATE BLACK BOX ALGORITHMS, JUST FEW AMONG MORE THAN 60 CRITICAL INHERENT DEFICIENCIES.
HUMANS HAVE THINGS A COMPUTER CAN NEVER HAVE.. A SUBCONSCIOUS BRAIN LOBE, REM SLEEP WHICH BACKS UP BETWEEN RIGHT/ LEFT BRAIN LOBES AND FROM AAKASHA BANK, A GUT WHICH INTUITS, 30 TRILLION BODY CELLS WHICH HOLD MEMORY, A VAGUS NERVE , AN AMYGDALA , 73% WATER IN BRAIN FOR MEMORY, 10 BILLION MILES ORGANIC DNA MOBIUS WIRING ETC.
SINGULARITY , MY ASS !
Generally, the Black Box Problem can be defined as an inability to fully understand an AI’s decision-making process and the inability to predict the AI’s decisions or outputs. However, whether an AI’s lack of transparency will have implications for intent and causation tests depends on the extent of this lack of transparency.
A complete lack of transparency will in most cases result in the complete failure of intent and causation tests to function, but some transparency may allow these tests to continue functioning, albeit to a limited extent.
Strong Black Boxes: Strong black boxes are AI with decision making processes that are entirely opaque to humans.
There is no way to determine (a) how the AI arrived at a decision or prediction, (b) what information is outcome determinative to the AI, or (c) to obtain a ranking of the variables processed by the AI in the order of their importance.
Importantly, this form of black box cannot even be analyzed ex post by reverse engineering the AI’s outputs.
Weak Black Boxes: The decision-making process of a weak black box are also opaque to humans. However, unlike the strong black box, weak black boxes can be reverse engineered or probed to determine a loose ranking of the importance of the variables the AI takes into account. This in turn may allow a limited and imprecise ability to predict how the model will make its decisions. .
Intent tests appear throughout the law and have developed over centuries to help courts and juries understand and regulate human conduct. Intent, for example, is a means of finding out whether a person intended to cause a particular outcome to occur. Intent may also determine whether the severity of a penalty is appropriate for the particular conduct.
Machines and computer programs have no intent. The most we can glean from how they work and how they are designed is what goals their users or creators sought to achieve and the means they permitted their machine or program to use to achieve them.
It makes sense to speak about the intent of the designer or user. For example, we may infer from a computer program designed to break into a computer system that its creator intended to use it for that purpose.
In some cases, we can look at the computer program’s instructions to determine what the designer of the program was trying to accomplish and what means could be used by the program to accomplish that goal..
Black-box AI, however, may function in a manner well outside of what the program’s creators could foresee. To be sure, we may be able to tell what the AI’s overarching goal was, but black-box AI may do things in ways the creators of the AI may not understand or be able to predict.
An AI securities-trading program, for example, may be given the overarching goal of maximizing profit, but how it makes its trading decisions or whether it meets its objective through market manipulation may be entirely unclear ex ante to its creators and even expost to courts and regulators.
Because we cannot look to the program’s instructions or design to determine the intent of its creator or user, intent tests become impossible to satisfy If intent tests cannot be satisfied, laws relying on them will cease to function. Most critically, because intent tests appear in the law where penalties are most severe (such as in criminal statutes), the most dangerous or noxious conduct may go unregulated if it is AI, rather than a human, that engages in it.
AI would be exempt from the most stringent aspects of our criminal, securities, and antitrust laws, which often require a showing of intent to give rise to the most serious forms of civil and criminal liability intent tests fail largely for three reasons:--
(1) An AI’s conduct or decisions may tell us nothing about its designer’s or user’s intent, which means that intent tests based on a person’s intent to achieve an unlawful outcome become unsatisfiable;
(2) Because it may be impossible to determine the bases of an AI’s decision or prediction, intent tests that scrutinize the bases or justifications for conduct become unsatisfiable; and
(3) Because intent tests often serve as a gatekeeper, limiting the scope of claims, they may entirely prevent certain claims or legal challenges from being raised when AI is involved.
All of these problems threaten to leave AI unregulated either because defendants that use AI may never be held liable (e.g., the government’s use of AI may prevent a showing of discriminatory intent) or claimants that rely on AI may be left without legal redress (e.g., because a plaintiff that uses AI to make investment decisions is unable to show reliance). .
In a neural network, for example, a series of interconnected artificial neurons may be mimicking data from the past, which may have simply reflected that a rise in price was correlated with placing and withdrawing trades. If the AI is a black box, there is no way of knowing.
While the author of the AI’s programming could have expressly prohibited this sort of conduct, the failure to do so is likely the result of negligence, not an intentional design decision. This would fall short of the sort of intent required for most criminal laws or civil fraud causes of action.
This problem becomes even more intractable where the particular basis for a decision is the central question of a litigation or regulatory investigation. For example, consider another hypothetical from the securities laws. A large financial institution uses AI to appraise homes that will serve as collateral for mortgage-backed loans that it will make and ultimately package into mortgage-backed securities.
The financial institution then provides the appraised values in its offering documents for the mortgage-backed security. Under this rubric, our hypothetical would almost never result in liability, even if the AI renders demonstrably problematic valuation opinions.
The financial institution can always say that it designed the AI with the utmost care and verified its accuracy on past data. It will almost always be able to argue that it subjectively believed in the valuation opinions it was publishing — it designed a highly sophisticated machine to carefully look at every house underlying every mortgage.
A plaintiff may be able to argue that the AI was not accurate enough for such reliance, that the AI was inadequately tested on out-ofsample data, and even that the issuer had some duty to sanity check the results, but none of these arguments will likely suffice to allege a subjective disbelief of the opinions.
There will have to be something more, particularly at the motion to dismiss stage, that would make plain that algorithm’s design or opinions were problematic and that the issuer knew it. Again, as with the Effect Intent cases, the opinion statement test will be actionable only at the margins — where there is particularly obvious and egregious conduct.
Black-box AI ensures that the problems will be impossible to detect without access to the AI and the ability to probe it to determine why it makes particular decisions. Even then, the AI may be a com- plete black box. It is almost impossible for a plaintiff, such as a purchaser of mortgage-backed securities, to be able to make allegations that the AI was designed or tested poorly or made decisions that put the user of the AI on alert that something was wrong.
Omission claims complicate things further. That is, a deeper problem occurs when courts require a detailed probe of the basis for an opinion. If the AI gave too little weight to a particular factor or piece of information, there would be no way of knowing it.
Even if the AI is a weak black box, the user of the AI may not be able to tell if a particular piece of information was outcome determinative. Causation tests also fail when black-box AI is involved. Most causation tests are used to limit the scope of far-reaching causes of action, such as ordinary negligence.
Doctrines such as proximate cause ensure that only reasonably foreseeable effects give rise to liability. Such a doctrine encourages individuals to act reasonably and penalizes those who do not. The proximate cause standard is thus a means of tying the scope of liability to the nature of the conduct at issue.
Other related doctrines, such as reliance, require the injured to prove that the harm they suffered was related to the allegedly unlawful conduct. Thus, a fraud claim will often fail unless the plaintiff can prove that the misrepresentation was something that the plaintiff took as true and that informed or caused the plaintiff to act to his detriment.
Loss causation, a doctrine that also appears in fraud-based claims, will also place similar limits on claims: only losses that stem from the alleged misrepresentation will be redressed
The inability to foresee harm is even greater with black-box AI because there is little or no ability to foresee how the AI will make decisions, let alone the effects of those decisions. If the AI is a weak black box, it may be possible to prove reliance because a loose ranking of the parameters fed to the model is available for analysis.
However, the investor would still face arguments that there is no way of proving that a particular piece of information would generally be outcome-determinative. it is not clear that certain forms of AI that are based on complex machine-learning algorithms, such as deep neural networks, will become more auditable and transparent in the future.
In fact, it may be that as these networks become more complex, they become correspondingly less transparent and difficult to audit and analyze. Indeed, commentators have speculated that AI may eventually become significantly more intelligent than human beings, such that they will surpass the analytical abilities of humans altogether. If this is the trajectory of AI, then it makes little or no sense to impose regulations requiring minimum levels of transparency.
It may be that certain technology may never meet the ideal levels of transparency desired by regulators and governments. If the improvement of AI requires, for example, more complexity that will cause a further lack of transparency, imposing transparency requirements will be tantamount to a prohibition on improvement or an invitation for companies to circumvent the rules.
Regulating the minimum levels of transparency for AI would at least implicitly be a regulation of design trade-offs. AI designers deciding whether to increase the size and depth of a neural network (thereby losing transparency) may be forced to use a shallower or less complex architecture to comply with regulations, even if such a design decision would result in poorer performance.
This essentially makes regulators and legislators arbiters of design — a function that regulators are not only less likely to be proficient at than AI developers, but are also reluctant to perform AI talent is concentrated in the hands of a few large firms.
Imposing the cost of compliance with a byzantine system of regulations would ensure that only large firms could afford to comply with them AI supervised by humans will pose the least problems for intent and causation tests, whereas autonomous AI will require liability schemes based on negligence, such as those used in agency law for the negligent hiring, training, or supervision of an agent.
When the AI operates under human supervision, the degree of transparency may shed light on the creator or user of the AI’s intent. When the AI is permitted to operate autonomously, the creator or user of the AI should be held liable for his negligence in deploying or testing the AI In the most dangerous settings, strict liability may be appropriate. The overall picture is a sliding scale of intent and foreseeability required for liability.
The Supervised Case --
AI that is supervised by a human is unlikely to pose significant problems for traditional intent and causation tests. For example, a human that consults AI to make decisions, such as a judge that consults AI to assist with sentencing decisions, ultimately makes decisions himself.
The AI may assist with the decision-making process, but the responsibility for the decision lies largely with the human decision-maker. In such a case, the transparency of the AI will inform intent and causation inquiries.
For example, if a human relies on AI that is fully transparent, then he can determine how the AI is making its decisions and will be able to foresee the effect of the AI’s decisions. Intent and causation tests will therefore properly apply because the foreseeable consequences of the AI’s decisions will be ascertainable.
When AI is a black box, the degree of transparency bears directly on the intent of a human who makes decisions based on the AI. For example, blind reliance on AI that engages in a decision-making process that the human cannot understand and that may have effects that the human cannot foresee may be evidence of unlawful intent such as scienter or willful blindness.
The extent to which the AI is a black box thus bears on the human’s intent. In such cases, courts and regulators will not need to look to the design of the AI or the foreseeable effects of the AI to determine liability.
The central question in such cases will be the degree of the AI’s transparency and the culpability or reasonableness of the human’s reliance on the black-box AI. AI is a new and unprecedented form of agent. When it operates autonomously, it is indistinguishable in some cases from a human being tasked with meeting some objective.
Just as a human may behave in an unpredictable manner, AI may also arrive at solutions or engage in conduct that its user or creator never foresaw, particularly when the AI is a black box. Notwithstanding the similarities between an AI and a human agent, a vicarious liability rule, such as respondeat superior, would make sense only in certain circumstances.
When the AI operates autonomously in a mission-critical setting or one that has a high possibility of externalizing the risk of failure on others, such as when it is used in a highly interconnected market or to perform a medical procedure, the AI’s user or creator should be more broadly liable for injury the AI inflicts, and a vicarious liability rule is appropriate.
In such cases, a lack of transparency should not insulate the user or creator of the AI from liability. Instead, the risks of deploying a blackbox AI autonomously in such settings should fall on the AI’s user or creator because the AI was used notwithstanding its unpredictable and impenetrable decision-making. In such a case, the imposition of vicarious liability would be functionally equivalent to a strict liability regime.
When the AI is deployed autonomously in less dangerous or mission-critical settings, a vicarious liability rule may be less appropriate.
There may be little risk of harm from the AI’s error in these circumstances and holding the user or creator of the AI liable regardless of intent or negligence would chill a large swath of desirable AI applications. Instead, in such cases, the negligent principal rule would be more appropriate.
When the AI is transparent, knowledge about how the AI’s decision-making process works may be used to establish the existence of or lack of reasonable care. When, however, the AI is a black box, the deployment of the AI in the face of a lack of transparency may be sufficient to establish a lack of reasonable care.
The question is similar to the one asked in the agency setting — whether the AI’s creator or user was negligent in deploying, testing, or operating the AI. The use of the AI in the face of a lack of transparency bears heavily on that question.
In the case where there is a lack of transparency, proximate cause tests should focus on the possible effects of deploying AI autonomously without understanding how it functions, rather than on the specific ability of the user or creator of the AI to have predicted the injurious effects of the AI’s conduct.
Consider the previous example of the AI that re-tweeted false or misleading information, was it reasonable for the creator of the AI to give the program the ability to create its own tweets? The focus in such a case would be on whether the risk of the potentially unlawful conduct was apparent or should have reasonably been addressed with precautions.
Putting the supervised and autonomous cases together, one can imagine four quadrants of liability. First, when there is both supervision of the AI and the AI is transparent, then the intent of the creator or user of the AI can be assessed through conventional means (i.e., fact-finding mechanisms such as depositions and subpoenas) as well as by examining the AI’s function and effect.
Second, when the AI is supervised but to some degree a black box, intent must be assessed based on whether the creator or user of the AI was justified in using the AI as he did — with limited insight into the AI’s decision-making or effect.
Third, if the AI is autonomous but supervised, the rule that should apply is the principal-supervision rule from agency law. The question will be whether the creator or user of the AI exercised reasonable care in monitoring, constraining, designing, testing, or deploying the AI. Fourth, when the AI is both autonomous and unsupervised, the sole question will be whether it was reasonable to have deployed such AI at all.
The answer may simply be no, which means that the creator or user of the AI would be liable for the AI’s effects, even if he could not foresee them and did not intend them Liability rules that employ Effect Intent or Gatekeeping Intent tests will require a threshold inquiry into the autonomy of the AI. In cases where the AI operates autonomously, the intent tests should be relaxed to allow for evidence of negligence.
In other words, specific intent should not be required for liability, nor should such tests be used to narrow the scope of potential claims when autonomous AI is involved. In cases where a basis for conduct must be articulated, such as in antitrust, securities, or constitutional cases, the use of black-box AI should be prima facie evidence that, apart from the past accuracy of the AI, the user or creator of the AI lacked a sufficient justification for a given course of conduct.
This can be accomplished through burden shifting. In other words, in cases involving autonomous AI that lack transparency, the burden should be on the proponent of the AI to prove that the AI’s conduct or decisions were justified. Such a test would encourage a human check on the AI’s decisions and conduct.
Proximate cause tests should assess the level of human supervision at the outset. It is only in the autonomous case that foreseeability of harm will be exceptionally difficult to assess. In such cases, the question should be focused on the foreseeability of harm from deploying AI autonomously given its degree of transparency.
Thus, the causation analysis for AI that falls into the Black Box / Less Supervision quadrant above should turn not on whether the particular harm caused by the AI was reasonably foreseeable, but whether the harm was a foreseeable consequence of deploying black-box AI autonomously.
The question is one of conceivability, not foreseeability in such settings.
For example, a heuristic for model-based interpretability is "simulatability": whether or not a human is able to internally simulate and reason about the model's entire decision-making process . Yet there is no way to consistently quantify this heuristic, or any of the potentially dozens of interpretability measures. The development of interpretable AI is on a random walk without metrics to guide and measure progress.
Yet more confounding is having to choose between model performance and interpretability, particularly in medical applications where transparency and trust are critical . Herein lies the second core issue: misalignment.
Perhaps when faced with a performance vs interpretation tradeoff, solutions can be found with interpretation interfaces and modules: use powerful black-box models, and layer-on an interpretation module for post-hoc inspection and explainability.
Causal relationships don’t happen by accident.
It might be tempting to associate two variables as “cause and effect.” . Extensively test the relationship between a dependent and an independent variable before asserting causality. False positive, is where a causal relationship seems to exist, but actually isn’t there.
Correlation is a relationship or connection between two variables where whenever one changes, the other is likely to also change. But a change in one variable doesn't cause the other to change. That's a correlation, but it's not causation
Causation without correlation is possible. It is well known that correlation does not prove causation. The upshot of these two facts is that, in general and without additional information, correlation reveals literally nothing about causation. .
Correlation is a statistical measure that indicates the extent to which two or more variables fluctuate together. A positive correlation indicates the extent to which those variables increase or decrease in parallel; a negative correlation indicates the extent to which one variable increases as the other decreases
As machine learning has expanded beyond its roots in the worlds of computer science and statistics into nearly every conceivable field, the data scientists and programmers building those models are increasingly detached from an understanding of how and why the models they are creating work.
To them, machine learning is akin to a black box in which you blindly feed different mixes of training data in one side, twirl some knobs and dials and repeat until you get results that seem to work well enough to throw into production.
Beyond the obvious issues that such models are extraordinary brittle, the larger issue is the way in which these models are being deployed. It is entirely reasonable to use machine learning algorithms to sift out extraordinarily nuanced patterns in large datasets.
Indeed, a very powerful application of machine learning can be around identifying all of the unexpected patterns underlying phenomena of interest in a dataset or to verify that expected patterns exist.
Where things go wrong is when we reach beyond these correlations towards implying causation.
Pattern verification is an especially powerful way of using machine learning models to both to confirm that they are picking up on theoretically suggested signals and, perhaps even more importantly, to understand the biases and nuances of the underlying data. Unrelated variables moving together can reveal a powerful and undiscovered new connection with strong predictive or explanatory power.
On the other hand, they could just as easily represent spurious statistical noise or a previously undetected bias in the data. Bias detection is all the more critical as we deploy machine learning systems in applications with real world impact using datasets we understand little about.
Perhaps the biggest issue with current machine learning trends, however, is our flawed tendency to interpret or describe the patterns captured in models as causative rather than correlations of unknown veracity, accuracy or impact.
Partially this is because often the thing we are most interested in cannot be directly observed through any single variable. Predicting most events, from the likelihood a user will buy a given product through the likelihood a given country will collapse into civil war tomorrow, relies on a patchwork of signals, none of which directly measure the actual thing we are interested in.
In essence, in most machine learning, the actual thing we hope to have our model learn cannot be learned directly from the data we are giving it.
This may be because the medium available (such as photographs) do not fully capture the phenomena we hope to have it recognize (such as identifying dogs). A pile of dog photographs cannot build a model to recognize their barks.
More often, it is because the thing we hope to measure (like the conversion of a website visitor into a customer) cannot be directly assessed through any single variable. Instead, we must proxy it through all sorts of unrelated variables that capture bits and pieces of the intangible “thing” we are trying to predict.
These bits and pieces, however predictive they may be, are merely genuinely or spuriously correlated with the thing we’re trying to predict. They do not necessarily cause or even explain how and why that thing occurs.
Moving from correlation to causation is especially important when it comes to understanding the conditions under which a machine learning model may fail, how long we can expect it to continue being predictive and how widely applicable it may be.
Using machine learning to identify correlative patterns in data is an extremely powerful approach to understanding both the nuances and biases of our data and the unexpected very real patterns that our current theoretical understandings failed to point us towards.
When we attempt to reach past this usage towards treating our models as “discovering” or “learning” causative new “natural laws” or concrete “facts” about the world, we tread upon dangerous ground.
Putting this all together, the ease with which modern machine learning pipelines can transform a pile of data into a predictive model without requiring an understanding of statistics or even programming has been a key driving force in its rapid expansion into industry.
At the same time, it has eroded the distinction between correlation and causation as the new generation of data scientists building and deploying these models conflate their predictive prowess with explanatory power.
In the end, as technology places ever more powerful tools in the hands of those without an understanding of how they work, we are creating great business and societal risk if we don’t find ways of building interfaces to these models such that they are able to communicate these distinctions and issues like data bias to their growing user community that lacks an awareness of those concerns
Today, an institution can take two approaches to reduce false alarms. It can either lower risk thresholds to capture more suspicious activities, or it can tighten risk thresholds to lower the number of false positives.
If an institution lowers its thresholds, the number of false positives increases; if it tightens its thresholds, the probability of missing fraud cases increases. With limited options, complex and changing obligations, and massive volumes of data to screen, the industry agrees that false positive rates are uncontrollable and compliance programs have become barely manageable.
A recursive neural network (RNN)—a type of deep learning AI—can dramatically increase the accuracy of threat detection, reducing or eliminating false positives. Unlike other technology solutions, machine learning can be tuned and trained to get better over time using the RNN
A true positive is an outcome where the model correctly predicts the positive class. Similarly, a true negative is an outcome where the model correctly predicts the negative class. A false positive is an outcome where the model incorrectly predicts the positive class.
The false positive rate is calculated as the ratio between the number of negative events wrongly categorized as positive (false positives) and the total number of actual negative events (regardless of classification)
in binary classification, a false positive is an error in data reporting in which a test result improperly indicates presence of a condition, such as a disease (the result is positive), when in reality it is not present, while a false negative is an error in which a test result
Interpretable Machine Learning models can have different levels of transparency. Three major levels of transparency achieved by an interpretable machine learning model are explained below.
Algorithmic Transparency bestows the user with the capability of understanding the inner processes by which the learning model generated its output given a certain input. A learning model is said to have achieved algorithmic transparency if the learning model can be investigated entirely by just mathematical models and analysis.
Decomposability in terms of transparency stands at a level higher than algorithmic transparency. Decomposability imparts the learning model with the capability of explaining its every part i.e. input, parameter and mapping to the user. Not every algorithmically transparent learning model is decomposable.
For a learning model to be decomposable, in addition to the restriction of algorithmic transparency i.e. able to be explored by mathematical models and analysis, it should also possess the potential that all of its variable, parameter and functions be comprehensible to the user without any external assistance.
For a learning model to be decomposable, in addition to the restriction of algorithmic transparency i.e. able to be explored by mathematical models and analysis, it should also possess the potential that all of its variable, parameter and functions be comprehensible to the user without any external assistance.
Simulatability deals with the capability of a learning model to be simulated by its users. Simulatability is the highest level of transparency a learning model can accomplish. Hence not all decomposable learning models are simulatable. For a decomposable learning model to possess the quality of simulatability, it should possess the ability to be thought or reasoned about as a whole in a self-contained manner without any additional help.
Machine Learning Interpretability is useful when the learning model is not interpretable by design and hence the user has to employ external interpretability techniques.
The interpretability technique to be used for a machine learning model’s interpretability is decided based upon 1) the users intention i.e. how the user wants the learning model to be explained (by text or visualizations), 2) the procedure to be used i.e. attention analysis and 3) the type of data accepted by the learning model.
The interpretability technique to be used for a machine learning model’s interpretability is decided based upon 1) the users intention i.e. how the user wants the learning model to be explained (by text or visualizations), 2) the procedure to be used i.e. attention analysis and 3) the type of data accepted by the learning model.
Post-hoc explainability techniques are majorly divided into the following.
Visual Explanations are the most effective method of explaining the machine learning model’s functioning to users with little expertise in machine learning. Most of the post-hoc explainability techniques involve dimensionality reduction so that the intricate mapping of machine learning model’s variables can be visualized on a 2D or 3D plot.
Local Explanations post-hoc explainability techniques deal with explaining the complex machine learning model by partitioning the solution space and then explaining the simpler solution subspaces that are pertinent for the whole machine learning model.
Hence local explanations work by decomposing the solution space and then explain the functioning of the machine learning model with respect to the decomposed solution subspace. If required these separate explanations can then be re-unified for full-scale machine learning model interpretability.
Hence local explanations work by decomposing the solution space and then explain the functioning of the machine learning model with respect to the decomposed solution subspace. If required these separate explanations can then be re-unified for full-scale machine learning model interpretability.
Explanations by simplification procedures attempt at creating a new simplified version of the machine learning model to be analyzed such that the new model resembles the functioning of the former model. Also, the new model should be less complex while having the same performance score. If all the above conditions are met for the new simplified machine learning model then it becomes easy to interpret the new model due to its less complexity.
Explanations by Text tackles the problem of explaining the results and outputs of the machine learning model by learning text explanations and symbols that portray the inner-working of the model. The idea behind generating symbols is to learn the semantic mapping from model to symbols to understand the reasoning of the machine learning algorithm.
Explanations by Example methodology involves drawing out specific class representative candidate examples with the intent of comprehending the mapping of variables in the machine learning model as well as the rationale of the algorithm. This method is similar to the process undertaken by a human while trying to understand a complex process.
Feature Relevance Explanations assigns appropriate importance or relevance scores to the variables of the machine learning model. The scores are allocated based on how a particular variable or parameter affects the output of the machine learning model.
AI has the ability to circulate tendentious opinions and false data that could poison public debates and even manipulate the opinions of millions of people, to the point of endangering the very institutions that guarantee peaceful civil coexistence.
If any major military power pushes ahead with AI weapon development, a global arms race is virtually inevitable, and the endpoint of this technological trajectory is obvious: autonomous weapons will become the Kalashnikovs of tomorrow.
Unlike nuclear weapons, they require no costly or hard-to-obtain raw materials, so they will become ubiquitous and cheap for all significant military powers to mass-produce. It will only be a matter of time until they appear on the black market and in the hands of terrorists, dictators wishing to better control their populace, warlords wishing to perpetrate ethnic cleansing, etc.
Autonomous weapons are ideal for tasks such as assassinations, destabilizing nations, subduing populations and selectively killing a particular ethnic group. We therefore believe that a military AI arms race would not be beneficial for humanity. There are many ways in which AI can make battlefields safer for humans, especially civilians, without creating new tools for killing people.
The creators of AI must seek the insights, experiences and concerns of people across ethnicities, genders, cultures and socio-economic groups, as well as those from other fields, such as economics, law, medicine, philosophy, history, sociology, communications, human-computer-interaction, psychology, and Science and Technology Studies (STS).
This collaboration should run throughout an application’s lifecycle — from the earliest stages of inception through to market introduction and as its usage scales
Professional human judgment is still necessary in AI to decide on the value of the information produced by the system and its uses in looking for material misstatements and financial fraud. In this regard, the acronym GIGO (“garbage in, garbage out”) may be appropriate.
Unless the data is reliably provided and processed, AI will produce results that are inaccurate, incomplete or incoherent, and machine learning would be compromised with respect to ethical AI.
Artificial intelligence (AI): the program that interprets and analyzes the data without explicit coding, makes predictions, and executes decisions to maximize a utility function (or goal). We limit our scope of AI to the subset of machine learning where programs learn to change when exposed to new data.
Internet of Things (IoT): provides raw data through the extension of enabled processors, sensors, and communications hardware into physical devices and everyday objects. IoT devices communicate through a shared access point or directly between other connected devices.
Big data analytics: the process of cleaning, structuring, and analyzing (mining) raw data to discern patterns, preferences, and trends. AI requires the raw data provided through analytics to learn how to make better decisions.
Explainable AI reduces the risks associated with regulatory and reputational accountability for safety and fairness.
Explainable AI technologies may identify privacy violation risk, with options for privacy-aware machine learning (PAML), multiparty computation, and variants of homomorphic encryption to identify privacy violation risks.
Training and education to develop the skills needed to mitigate risks in black-box models.
This should include:--
How to make data science and ML models interpretable by design and select the right model transparency from a range of models, from least to most transparent.
How to select the right model accuracy when required, and methods of validating and explaining these models.
Various methods, such as generative explainability and combining simple, but explainable models, with more complex, but less explainable ones.
Exploring the latest explainability techniques, such as the ones that are tracked by DARPA, or that are coming from commercial vendors.
Techniques for understanding and validating the most-complex types of predictive models.
Communication and empathy skills for data scientists to detect the users’ attitude and needs for explainability and successful AI adoption.
Establish AI ethics boards and other groups that are responsible for AI safety, fairness and ethics. These boards should include internal and external individuals known for their high reputation and integrity.
Over the last decade, 93% of European banks have been fined for AML-related offences; globally, banks have been fined approximately $31 billion over the last 10 years. Most of the fraud is deliberate.
Financial services have understandably not taken full advantage of AI solutions because of concerns with the so-called “black box” models, i.e., the model performs functions that are not transparent to the end user. If the bank does not understand how its technology is monitoring for financial crime, it cannot explain how it is complying with regulations to its regulators.
Models using AI, no matter how intelligent, cannot be expected to operate without human oversight and testing.
Even the best screening systems produce a high-rate of false positives that must be dispositioned by a human reviewer, by either clearing the alert, or escalating it for further review.
AI systems are capable of performing link analysis, drawing inferences by identifying entities that are parties to suspect transactions. criminals are constantly developing new methods of hiding their activity. AI can be used to identify new behaviours that would alert the financial institution to investigate.
One of the AI techniques to do so is called intelligent segmentation
Segmentation is a fundamental component of the anti-money laundering (AML) process, and is concerned with the groupings of entities based on similar business attributes and transactional behavior.
Segmentation, when done well, enables AML typologies to focus on unusual behavior for specific groups of entities, using thresholds that allow precise detection of bad actors while minimizing the number of false positive alerts. In a large, geographically distributed bank that provides correspondent banking services, the transactions that involve non-bank customers appear as 3rd party corresponds, also known as pseudo-customers.
Unlike a bank’s own customers for which KYC (know your customer) information is available, very little is known about these pseudo-customers. Because a definite identification of the party does not exist, it is particularly difficult to monitor such customers and categorize them.
Most institutions employ a hierarchical approach to segmentation. This approach requires business attributes for the top-down analysis and transaction summaries for the bottom-up analysis. This doesn’t work for pseudo-customers.
Due to the absence of KYC information, the business attributes for non-bank customers are very limited, meaning the top-down analysis is not practical. Moreover, the bottom-up analysis is only focused on the rough summary of transactional behaviors, such as total transaction volume and/or dollar amount. The result is large, uneven segments that lack defining characteristics.
This is particularly problematic from a model acceptance perspective. The inadequacy of explanatory features precludes the business user (or internal model review board) from approving the model.
Even if the segmentation could be implemented, the segmentation is ultimately static. What that means is that the segmentation model cannot adapt to the inevitable changes we see in behavior. An inaccurate segmentation model with large, uneven groups leads to high alert volumes with high level of noise (false positives) translating to higher investigator FTE requirements.
In order to capture the non-bank customers’ transactional behavior, an exhaustive list of features needs to be created in order to uncover the hidden behavior and reflect the AML risks. Some sophisticated institutions have turned to machine learning algorithms such as K-Means Clustering to solve for this.
This, however, requires rigorous assumptions about the distribution of the underlying data (the n of K). This approach is non-performant in the face of high-dimensional problems – which is exactly what defines the pseudo-customer problem.
With traditional monitoring systems, banks typically segment their customers by their industry, the type of business, size, as well as other factors. They apply rules that have worked historically for businesses in those segments.
The problem with this approach is that these segments do not consistently represent groups of entities with consistent transaction behaviour. it analysed transactions, observed patterns, and created new and more relevant segments, placing customers in them based on their behaviour.
A segment, for instance, might include entities that engage in large wire credit transactions, have high-frequency counterparties, and a large number of unique originators.
If a customer executed transactions that were outside of the normal parameters for their segment, they would be subject to further analysis, including, potentially, investigations by humans.
Fraudsters can attempt to take over the bank accounts of unsuspecting customers or clients and transfer money directly to them. An anomaly detection-based solution that could identify changes in customer behavior and analyze them for patterns related to fraudulent money transfers.
False positives could result in more work for the rest of the team fighting money laundering alerts. Individual banks may not require a perfect false positive rate. However, it is in the best interest of vendors to offer a service that could discern between fraudulent payments that appear legitimate, and legitimate payments that appear fraudulent. their solution could do this by recognizing potentially fraudulent patterns in customer behavior as they develop over time
Desirable core capabilities:--
Auto Feature Engineering: Detecting data points within transactions and customer behavior with a high potential fraud likelihood.
Intelligent Segmentation: Benchmarking certain behavioral patterns based on the transaction history and real time behavior of the client’s customer base.
Behavioral Insights: Tracking daily changes in customer behavior as well as which customers have deviated the most.
Intelligent Event Triage: Determining whether a given deviation in customer behavior is acceptable or if it needs to be investigated further.
Contextual Alert Information: Automatically providing the correct information to allow human employees to escalate an alert to investigators.
The software could exclusively recognize patterns that are potentially harmful and not acceptable deviations in transaction data. These types of patterns could serve as early indicators of money laundering, and some clients may find it important to keep track of them as they emerge.
Intelligent segmentation is A way of exploring how to divide up a customer base so that the machine learning model could analyze a sizeable amount of data at a time. This is done by creating thresholds of customer behavior based on the rate of deviation from each customer’s typical behavior.
Each threshold carries a different level of priority, which allows client companies to focus on accounts that have seen the most deviation first. this could also help their clients reduce false positives because more specific customer segments could lead to a more thorough analysis.
Behavior metrics could include how far each customer deviates from their normal behavior over time, or how that rate compares to other customers. This comparison could be scaled to only cover customers in the same segment or could be brought out to the entire customer base.
Using the behavioral insights interface, analysts could organize transactions showing behavioral deviations based on how much their patterns have changed. They could also organize them according to how much each transaction is worth and compare that to the rate of deviation.
This likely helps users close in on individual money laundering attempts as one could track the dollar amounts as they rise along with a rise in transactions with new or suspicious entities.
AML MUST come with an event triage tool that allows for identification of the most important information for investigators to use or analyze in the future. This purportedly results in a smaller amount of events extracted from transactional data than other solutions. That said, these results are purported to have a higher chance of being used to discover money laundering or indicative behavior.
Money laundering investigators need sufficient context on an alert in order to be able to track down threats. If there is no data that can serve as evidence of an attempt at a fraudulent transaction, an investigator would not be able to find a relevant starting place to do their job. This information might include the ID numbers and account history of the affected accounts, and the exact time the alert occurred.
AML solution provides this context which also allows them to make faster decisions. Each alert purportedly comes with a visual representation of the behavioral pattern that the solution marked as suspicious. the software supplies relevant data regarding this suspicious behavior so that investigators have an audit trail to work from
Fraudulent activities in finance can be detected by looking at on-surface and evident signals. Unusually, large transactions or the ones that happen in atypical locations obviously deserve additional verification.
Machine learning allows for creating algorithms that process large datasets with many variables and help find these hidden correlations between user behavior and the likelihood of fraudulent actions
MasterCard integrated machine learning and AI to track and process such variables as transaction size, location, time, device, and purchase data. The system assesses account behavior in each operation and provides real-time judgment on whether a transaction is fraudulent. The project aims at reducing the number of false declines in merchant payments
Machine learning algorithms analyze files written by insurance agents, police, and clients, searching for inconsistencies in provided evidence. There are many hidden clues in these textual datasets. The rule-based engines don’t catch the suspicious correlations in textual data, and fraud analysts can easily miss important evidence in boring investigation files. That’s why analyzing claims is one of the most promising spheres for machine learning applications.
Smart ML-backed algorithms are also efficient in duplicate claims detection or inconsistencies in car repair cost. Classifying data in repair claims solves the problem by uncovering hidden correlations in claim records or even behaviors of insurance agents, repair services, and clients
Healthcare and medical insurance is a rich area for fraud schemes due to a complex and bureaucratic process, which requires many approvals, verifications, and other paperwork. The most common scams are fake claims that use false or invalid social security numbers, claims duplication, billing for medically unnecessary tests, fake diagnosis, etc. Multiple data analytics approaches can mitigate such fraud risks.
Benford's law, also called the Newcomb–Benford law, the law of anomalous numbers, or the first-digit law, is an observation about the frequency distribution of leading digits in many real-life sets of numerical data. The law states that in many naturally occurring collections of numbers, the leading significant digit is likely to be small.
For example, in sets that obey the law, the number 1 appears as the leading significant digit about 30% of the time, while 9 appears as the leading significant digit less than 5% of the time. If the digits were distributed uniformly, they would each occur about 11.1% of the time
Upcoding is fraudulent medical billing in which a bill sent for a health service is more expensive than it should have been based on the service that was performed.
Upcoding and unbundling are methods of healthcare billing fraud involving the improper application of codes for medical diagnoses and procedures. ... When these healthcare providers and facilities improperly code the medical services they've provided in order to receive higher reimbursements, they commit coding fraud.
In 2013, Goldberg Kohn filed a lawsuit against IPC The Hospitalist Co. Inc. (IPC) – one of the largest providers of hospitalist services in the country. The lawsuit alleged that IPC submitted false claims to federal health care programs, such as Medicare and Medicaid. According to the lawsuit, IPC did so by encouraging its doctors to bill at the highest levels available, regardless of the level of service provided. The government decided to intervene in the matter. In 2017, TeamHealth Holdings, which had acquired IPC in 2015, agreed to pay $60 million under the settlement agreement.
Customers’ choices on marketplaces ( say Amazon ) are often driven by reviews. Some fraudsters create fake reviews for their accounts to attract customers or giving bad reviews to sabotage better products. Machine learning algorithms can eliminate the influence of such fraudsters through conducting sentimental and behavior analytics and detecting suspicious activities linked to merchants or their products.
Payments are the most digitalized part of the financial industry, which makes them particularly vulnerable to digital fraudulent activities. The rise of mobile payments and the competition for the best customer experience nudge banks to reduce the number of verification stages. This leads to lower efficiency of the rule-based approach. So, banks and payment companies switch to data analytics, machine learning, and AI-driven methods.
Modern fraud detection systems solve a wide range of analytical problems to uncover all scams in the payments streams.
Data credibility assessment. Gap analytics help identify missing values in sequences of transactions. Machine learning algorithms can reconcile paper documents and system data eliminating the human factor. This ensures data credibility by finding gaps in it and verifying personal details via public sources and transactions history.
Duplicate transactions. A common scam method is creating transactions close to original or making a copy of a transaction. For instance, a company tries to charge a counterpart twice with the same invoice by sending it to different branches.
Rule-based systems currently used constantly fail to distinguish between error or unusual transactions from real fraud. For example, a customer can accidentally push a submission button twice or simply decide to buy twice more goods. The system should differentiate suspicious duplicates from human errors.
While duplicate testing can be implemented by conventional methods, machine learning approaches will increase accuracy in distinguishing erroneous duplicates from fraud attempts.
Account theft and unusual transactions. Much of fraud detection in payments is focused on user behavior analysis during transactions.
For example, a client visits a specific supermarket at 9-10 pm every night. It’s located near the client’s house. The payment sum varies from $10 to $40. Every two days the client also drives to a gas station.
Once a transaction occurs in a different part of town in a bar and the sum is $40, the algorithm will consider this activity suspicious and assign a higher level of fraud likelihood. To check this transaction, the system will send a verification request to a card owner.
Descriptive stats like averages, standard deviations, and high/low values are very useful for analyzing behavior. These metrics allow for comparing separate transactions with personal or intra-group benchmarks. Payments with large standard deviations look suspicious. So, a good practice is to send a request for verification to an account owner if such deviations occur.
Preventing loan application fraud
Lending is sensitive to scams that abuse personal information. Just a decade ago, it was difficult for fraudsters to get access to IDs, photos, addresses, and mobile phone numbers. Today, nearly all data can be found in social networks or elsewhere on the Internet. This makes the life of financial institutions more difficult. Fraudsters become smarter, and loan applications require more rigorous assessment, while clients want to get money as soon as possible.
Personal details counterfeiting. A common type of fraud is often based on providing false personal information. Scammers provide personal details with a number of misspelling issues or misrepresentations of income or credit qualifications. It makes debt collection more difficult. The problem can be solved in two ways.
The first one entails reviewing the customer relationship history with a bank looking for inconsistencies and quickly verifying record fields via open APIs.
The second path is more sophisticated and requires building a scoring model or calculating fraud probability. Scoring models calculate the fraud possibility of the record and grade it against a standard scale. It helps assess which applications are more likely to be fraudulent. Machine learning and advanced analytics solve the problem of fraud probability assessment by classifying applications into groups.
Such solutions allow for minimizing costs by reducing the need to verify each application and concentrating the efforts on the risky loans only. It also improves general credit scoring – the process of grading customer creditworthiness – by distinguishing fraudsters from bad borrowers. Drawing a distinct line between fraudsters and problematic borrowers also ensures better credit statistics.
Machine learning for anti-money laundering
Regulators, banks, and investment firms are often involved in monitoring possible money laundering activity: They must detect and inform each other about suspicious activities.
Anomaly detection to reveal suspicious transactions
Anomaly detection is one of the common anti-fraud approaches in data science. It is based on classifying all objects in the available data into two groups: normal distribution and outliers. Outliers, in this case, are the objects (e.g. transactions) that deviate from normal ones and are considered potentially fraudulent.
The variables in data that can be used for fraud detection are numerous. They range from transaction details to images and unstructured texts.
By analyzing these parameters, anomaly detection algorithms can answer the following questions:
Do clients access services in an expected way?
Are user actions normal?
Are transactions typical?
Are there any inconsistencies in the information provided by users?
The anomaly detection approach is perhaps the most straightforward as it provides simple binary answers. This may be helpful in some cases. For instance, if the transaction looks suspicious, the system may ask a user to make multiple additional verification steps. Traditional anomaly detection doesn’t allow for revealing fraud, although it may be a good supportive instrument for existing rule-based systems.
But there are more advanced approaches that combine several ML algorithms to reduce uncertainty. They can be implemented using multiple machine learning styles and underlying mathematical models. Let’s have a look at the ones that are most common.
Advanced systems aren’t limited to finding anomalies but, in many cases, can recognize existing patterns that signal specific fraud scenarios. There are two types of machine learning approaches that are commonly used in anti-fraud systems: unsupervised and supervised machine learning. They can be used independently or be combined to build more sophisticated anomaly detection algorithms.
Supervised learning entails training an algorithm using labeled historical data. In this case, existing datasets already have target variables marked, and the goal of training is to make the system predict these variables in future data.
Unsupervised learning models process unlabeled data and classify it into different clusters detecting hidden relations between variables in data items
Labeling data. While data labeling can be done manually, it’s hard for humans to classify new and sophisticated fraud attempts by their implicit similarities. That’s why data scientists apply unsupervised learning models to segment data items into clusters accounting for all hidden correlations. This makes data labeling more precise: Not only does the dataset have labeled fraud/non-fraud items, but these labels also nuance different types of fraudulent activities.
Training a supervised model. Once the data is labeled, the next iteration is to apply this new labeled dataset to train supervised models that will be detecting fraudulent transactions in production use.
Ensembling multiple different models is a common approach in data science. While you can make a single model, it will always have its strengths and weaknesses. It will recognize some patterns, but miss the others.
To make predictions more accurate data scientists usually build multiple models using the same method (we’ll talk about methods in the next section) or combine entirely different methods. Thus, all models from the ensemble analyze the same transaction and then “vote” to make a final decision. It allows for leveraging the strengths of multiple different methods and make decision as precise as possible
Setting an express verification. Using ensembles requires a great deal of computing power and time to crunch all data. If you were to check all transactions, the time spent on calculations may harm user experience. That’s why a good practice is to make verification in two steps.
The express verification implies simple anomaly detection or another straightforward ML method to divide all transactions into regular and suspicious ones. As regular transactions don’t require further verification, the system approves them. Those that look suspicious are sent for advanced verification through a complex ensemble.
Supervised fraud detection method
In terms of actual machine learning methods, there are five commonly used types. We’ll cover only supervised learning methods as they can be applied in building complex ensembles.
Random forest (or an ensemble of decision trees) is an algorithm which builds decision trees to classify the data objects. The model selects a variable that enables the best splitting of records and repeats the splitting process multiple times. As a result, if we were to visualize how the algorithm works, the image would look like a tree. To make predictions more precise, data scientists train multiple decision trees on random subsets from a general dataset. To decide whether transaction looks like a fraud, trees vote, and the model provides a consensus judgment.
Random forests are relatively simple systems that you can use to set up fraud detection fast. The most common use case is payments systems.
Pros: Besides their simplicity and speed, random forests can be used with different types of data, including credit card numbers, dates, IP addresses, postal codes, etc. They are considered precise predictors that can work even with datasets that have missing records.
Cons: Sometimes engineers stumble over the problems with overfitting. Overfitting means that the model over-remembers the patterns in the training dataset and fails to make predictions on future data. Another problem is the dataset balance. If a dataset contains mostly normal transactions and just a small fraction of fraudulent ones, the accuracy may decrease.
Support vector machine
A support vector machine (SVM) is a supervised machine learning model that uses a non-probabilistic binary linear classifier to group records in a dataset. What does it mean? The algorithm divides data into two categories with a clear gap. The division line is defined by making several hyperplanes in the multidimensional space. Then the algorithm selects the hyperplane which separates records better than the other ones.
In machine learning, support-vector machines (SVMs, also support-vector networks are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis.
Given a set of training examples, each marked as belonging to one or the other of two categories, an SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier (although methods such as Platt scaling exist to use SVM in a probabilistic classification setting).
A SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on the side of the gap on which they fall.
SVM can be inferior to random forests in credit card transactions with small datasets, but can also approach their accuracy once datasets are large enough.
Pros: Support vector machines are particularly good at working with complex multidimensional systems. They also allow for avoiding the overfitting problem that random forests may experience. Generally, SVM is a very common method in credit card fraud detection. And the abundance of research work makes adjusting SVM-models for credit card fraud detection simpler for a data science team.
Cons: The complexity of SVM models will require much engineering effort to fine tune the algorithm and achieve high accuracy. Also, SVMs are very slow and computationally heavy. That’s why they will require powerful computing architecture.
K-Nearest Neighbor is an algorithm which classifies records by similarity based on the distance in multidimensional space. The record is assigned to the class of the nearest neighbors. The record of each cluster is voting for each new record using the distance parameter.
K-nearest neighbors is another common approach used to analyze credit card transactions.
Pros: The method is insensitive to missing and noisy data, which allows for configuring larger datasets with less preparation. It’s also considered highly accurate and doesn’t require much engineering effort to tweak models.
Cons: Like neural networks, k-nearest neighbors require powerful infrastructures and they also lack interpretability.
Neural networks and Deep Neural Networks
Neural Network is a model that allows for determining non-linear relations between the records. The algorithm structure is built on principles close to those of the human brain neurons. The model is trained on a labeled dataset making input data pass through several layers (i.e. sets of mathematical functions). The models of this type employ 1-2 hidden layers.
Deep Neural Networks works similar to neural networks but employs much more layers than a usual Neural Network. This provides more accurate results, as well as requires more computing power and time for data processing.
Deep learning has created a revolution in data science over the past years. Consequently, it also impacted the financial services industry. Currently, neural networks are being applied both for transactional verification and insurance claims.
Pros: Neural networks and especially deep neural networks are powerful at finding non-linear and very complex relations in large datasets. This works both for transactional data and for text and image analysis, which may be used in insurance cases. They usually provide high accuracy, which makes neural networks a necessary part of a modern fraud detection ensemble.
Cons: Neural networks are state-of-the-art systems that are very difficult to build and tweak to reach efficiency. They require highly skilled professionals and a powerful computing architecture. For this reason, we don’t recommend using the method for express analysis of all transactions. Another major problem with deep neural networks is the lack of interpretability.
While they may be highly accurate, it’s nearly impossible to define how specifically the system arrived at one conclusion or the other ..
Antifraud systems should meet the following standards:--
detect fraud in real-time
improve data credibility
analyze user behavior
uncover hidden correlations
While these qualities can be offered by machine learning algorithms, they have two serious drawbacks to be aware of. They still require large and carefully prepared datasets for training and still need some features of rule-based engines, like checking legal limitations for cash transactions.
Also, machine learning solutions usually require substantial data science skills to build complex and robust ensemble algorithms. This sets a high barrier for small and medium companies to use the technique leveraging internal talent. The task requires deep technological and domain expertise.
Descriptive Analytics, which use data aggregation and data mining to provide insight into the past and answer: “What has happened?”
Predictive Analytics, which use statistical models and forecasting techniques to understand the future and answer: “What could happen?”
Prescriptive Analytics, which use optimization and simulation algorithms to advise on possible outcomes and answer: “What should we do?”
Descriptive Analytics tells you what happened in the past. Diagnostic Analytics helps you understand why something happened in the past. Predictive Analytics predicts what is most likely to happen in the future. Prescriptive Analytics recommends actions you can take to affect those outcomes.
Descriptive analytics uses data aggregation and data mining of historical data to answer the question, “What happened?” Descriptive analytics is essentially the same as the science of statistics, providing information without analysis or insight.
The easiest way to define it is the process of gathering and interpreting data to describe what has occurred. Descriptive analytics takes the raw data and, through data aggregation or data mining, provides valuable insights into the past
Prescriptive analytics leverages both existing data and action/feedback data to guide the decision maker towards a desired outcome. Prescriptive analytics is also predictive in nature since it tries to estimate multiple futures based on your actions and advise on the outcomes before you actually make a decision.
Prescriptive analytics has two levels of human intervention: decision support, e.g. providing recommendations; decision automation, e.g. implementing the prescribed action . The effectiveness of the prescriptions depends on how well these models incorporate a combination of structured and unstructured data, represent the domain under study and capture impacts of decisions being analyzed
Apart from answering the “what will happen if…?” question, prescriptive analytics also tackles the “what should we do to reach the desired outcome?” component.
Prescriptive analytics, describes a type of analytics designed to answer the question, “What do we do now?” In prescriptive analytics, the outcome is not just a prediction or forecast, but recommendations for the best course of action.
Reducing risk in insurance and financial services
Detecting credit card fraud
Allowing more accurate predictions of supply and demand
Identifying threats and issues affecting computer networks
Predictive analytics in banking and financial services:--
Predictive analytics is valuable across the spectrum of banking and financial service activities, from assessing risk to maximizing customer relationships. Predictive analytics are used to:--
Prevent credit card fraud by flagging unusual transactions.
Scoring credit and deciding whether to approve a loan or credit applications.
Predict customer churn, allowing banks to reach out right before a customer is likely to switch institutions.
Predictive analytics and other emerging technologies
Predictive analytics is often conflated with other developing data and analytics technologies. Three technologies often confused with predictive analytics are machine learning, predictive modeling, and data mining.
Is predictive analytics the same as machine learning? Predictive analytics is not the same as machine learning. Machine learning, which allows computers to learn from their own activities, is one of the elements that can be applied as part of the predictive analytics process.
What are the most common models used in predictive analytics?
The most common models used in predictive analytics include linear regression, logistic regression, linear discriminant analysis, decision trees, naive bayes, K-nearest neighbors, support vector machines, random forest and boosting.
A more complete description of each is included below.
What is the relationship between how many sales leads a marketing campaign generates to the amount of money spent promoting the campaign?
How many more leads could be captured if the promotional budget was increased by, say, $10,000?
How much will the cost of raw materials used in manufacturing increase in a year?
Logistic regression. This compares a dependent variable with one or more independent variables to determine the probability of a particular outcome. Logistic regression could be used to predict how likely a person is to develop diabetes based on their age, sex, body mass, blood test results and family history, or which candidate in an election will most appeal to people with a particular combination of demographic information, such as age, race, income and location.
Linear discriminant analysis is used for classification. A typical example might be, “Based on answers to a survey, which group of customers is more likely to buy a particular product?”
Decision trees are binary, relying on yes/no questions to arrive at the outcome. A decision tree could be used to sort applicants for a job, for instance. Does the applicant have a college degree? If no, does the applicant have alternative qualifications? If yes, does the applicant have more than three years experience? If yes, does the applicant have a defined set of skills and experience?
Jewish health insurance companies wants India to kill off patients from whom they cannot milk profits .. and they have been arm twisting Bharatmata to make euthanasia a lawful thing ..
The condition for Jewish health insurance companies to come to India is euthanasia --as this is paramount for them to make humongous profits..... Modis health care is like Obama health care-- sole intention is to dish out massive profits to kosher health companies ... who are also political donors..
Kosher health companies bribe politicians and doctors in all western nations they thrive . People who take good care of their health ( by organic diet/ Yoga etc ) are forced to pay for drug addicts, sexual deviants and processed food consumers who abuse their health..
Euthanasia has to be decided by the elected law makers... not some collegium judges in deep state payroll...
Laws against euthanasia and assisted suicide are in place to prevent abuse and to protect people from unscrupulous doctors and others ( grab property )... Legalizing euthanasia would send a clear message: it is better to be dead than sick or disabled...
The patient is depressed and believes that his situation is worse than it really is.( or made to feel by selfish relatives )... Faced with a doctor who both heals and kills, the patient lives with an uncertainty that adds to vulnerability...
Patients who are ill or dependent often feel worthless and a burden to their family and loved ones. .. With legalized assisted suicide, the rate of non-assisted suicide WILL increase...
Every patient has the right to refuse treatment or to request that ongoing treatment be stopped... Life in society is based on relationships of mutual trust among all citizens. Everyone must be sure that nobody will kill him... .
We are already witnessing worrisome abuse in jurisdictions where euthanasia or assisted suicide is legal...
WE THE PEOPLE WILL NOT ALLOW HEALTH INSURANCE TO DECIDE WHO HAS THE RIGHT TO LIVE... RIGHT TO LIVE ALSO INCLUDES RIGHT TO DIE .. TEE HEEEEEEE ..
WHERE IS THE FUCKIN' INTELLIGENCE .. RIGHT TO HEAL ALSO INCLUDES RIGHT TO KILL? ..
The right to life does not create a right to choose death rather than life.It meant there was no right to die at the hands of a third person or with the assistance of a public authority.. The right to life is a moral principle based on the belief that a human being has the right to live and, in particular, should not be killed by another human being.
International law only allows law enforcement officers to deliberately take life (shooting to kill) where absolutely necessary to defend themselves and others against an imminent threat to life. .. The entitlement of a person to make the decision to end their own life through euthanasia is right to choose.
Passive euthanasia is a condition where there is withdrawal of medical treatment with the deliberate intention to hasten the death of a terminally-ill patient..
WHEN A MAN IS SICK , AND IS IN PAIN -- HIS MIND IS VULNERABLE.. GREEDY CHILDREN WILL KILL HIM OFF TO GRAB PROPERTY.
Disparate impact in United States labor law refers to practices in employment, housing, and other areas that adversely affect one group of people of a protected characteristic more than another, even though rules applied by employers or landlords are formally neutral.
Although the protected classes vary by statute, most federal civil rights laws protect based on race, color, religion, national origin, and sex as protected traits, and some laws include disability status and other traits as well.
In rule-based systems, the system is programmed to apply a decision-making tree. The questions asked and the path to a particular outcome, depending on the answers given, can be depicted by way of flow-chart (even if that flow-chart might be very large, involving numerous branches).
In contrast, statistical machine learning involves a computer system training itself to spot patterns and correlations in data sets, and to make predictions based on those patterns and correlations. The computer system is first trained on data sets provided by the system designer. Once trained, it can be used to infer information and make predictions based on new data.
A useful distinction is between decision making that is fully automated, where the algorithm makes the final decision, and decision making where there is a “human in the loop”, who uses the algorithm’s output to support their decision-making. This has implications not only under the GDPR (see Article 22) but also for the application of public law principles.
Even though there is a human in the loop, an individual may place over-reliance on the outcome of an automated-system, in effect simply rubber-stamping the decision.
Intentional opacity is where an algorithm is designed so that its workings are concealed, in order to protect intellectual property. Illiterate opacity arises where an algorithm is so complex that it is understandable only to tech experts.
Finally, intrinsic opacity is where a machine-learning system is so complex that even a tech expert is unable to understand its internal workings. That is, the system is a “black box”. A lack of transparency in decision making is clearly a primary hurdle to holding decision-makers to account where algorithms have been deployed.
Neural networks excel at churning through vast quantities of training data and making connections, absorbing the underlying patterns or logic for the system in hidden layers of linear algebra
They are terrible, however, at explaining the underlying logic behind the relationships that they’ve found: there is often little more than a string of numbers, the statistical “weights” between the layers. They struggle to distinguish between correlation and causation.
The dream of big data in medicine is to feed a neural network on “huge troves of health data, finding complex, implicit relationships and making individualized assessments for patients.” What if, inevitably, such an algorithm proves to be unreasonably effective at diagnosing a medical condition or prescribing a treatment, but you have no scientific understanding of how this link actually works?
The statistical models that underlie such neural networks often assume that variables are independent of each other, but in a complex, interacting system like the human body, this is not always the case.
Transparency about how the algorithm functions—the data it looks at, and the thresholds for drawing conclusions or providing medical advice—may be required, but could also conflict with the profit motive and the desire for secrecy in healthcare startups..
Artificial general intelligence, or AGI can be defined as the ability of a machine to perform any task that a human can.
Capt Ajit Vadakayil says—you cannot achieve AGI unless we use water as memory and Mobius coil organic DNA wiring..
Artificial general intelligence systems are designed with the human brain as their reference. Since we ourselves don’t have the comprehensive knowledge of our brains and its functioning, it is hard to model it and replicate it working.
Whereas AI is preprogrammed to carry out a task that a human can but more efficiently, artificial general intelligence (AGI) expects the machine to be just as smart as a human.
With the evolution of Internet of Things (IoT), there is a huge amount of data exchange among different smart devices located geographically apart from each other. As the data among these devices travels using an open channel, i.e., Internet, so one of the challenge is to build a secure system to protect from various cyber-threats.
Cyber-criminals use new attack vectors to launch various attacks as the attack surface is also increasing exponentially. To deal with these threats, the network requires traffic surveillance along with strong access control policies
Network intrusion detection systems (NIDS) are set up at a planned point within the network to examine traffic from all devices on the network. It performs an observation of passing traffic on the entire subnet and matches the traffic that is passed on the subnets to the collection of known attacks.
A network-based intrusion detection system (NIDS) monitors and analyzes network traffic for suspicious behavior and real threats with the help of NIDS sensors. It scrutinizes the content and header information of all packets moving across the network.
The NIDS sensors are placed at crucial points in the network to inspect traffic from all devices on the network. For instance, NIDS sensors are installed on the subnet where firewalls are located to detect Denial of Service (DoS) and other such attacks.
If your network is penetrated by a malicious attacker, it can lead to massive losses for your company, including potential downtime, data breaches, and loss of customer trust.
An intrusion detection system (IDS) is a tool or software that works with your network to keep it secure and flag when somebody is trying to break into your system.
An intrusion detection system (IDS) is a device or software application that monitors a network or systems for malicious activity or policy violations. Any intrusion activity or violation is typically reported either to an administrator or collected centrally using a security information and event management (SIEM) system. A SIEM system combines outputs from multiple sources and uses alarm filtering techniques to distinguish malicious activity from false alarms
IDS types range in scope from single computers to large networks. The most common classifications are network intrusion detection systems (NIDS) and host-based intrusion detection systems (HIDS).
A host-based intrusion detection system (HIDS) monitors and analyzes system configuration and application activity for devices running on the enterprise network. The HIDS sensors can be installed on any device, regardless of whether it’s a desktop PC or a server.
HIDS sensors essentially take a snapshot of existing system files and compare them with previous snapshots. They look for unexpected changes, such as overwriting, deletion and access to certain ports. Consequently, alerts are sent to administrators to investigate activities that seem iffy.
They are a highly effective tool against insider threats. HIDS can identify file permission changes and unusual client-server requests, which generally tends to be a perfect concoction for internal attacks. That’s why it should come as no surprise that HIDS is often used for mission-critical machines that are not expected to change.
A system that monitors important operating system files is an example of an HIDS, while a system that analyzes incoming network traffic is an example of an NIDS. It is also possible to classify IDS by detection approach.
The most well-known variants are signature-based detection (recognizing bad patterns, such as malware) and anomaly-based detection (detecting deviations from a model of "good" traffic, which often relies on machine learning). Another common variant is reputation-based detection (recognizing the potential threat according to the reputation scores).
Some IDS products have the ability to respond to detected intrusions. Systems with response capabilities are typically referred to as an intrusion prevention system. Intrusion detection systems can also serve specific purposes by augmenting them with custom tools, such as using a honeypot to attract and characterize malicious traffic.
NIDS vs. HIDS: What’s the Difference?
Each of these intrusion detection systems come with their own strengths. NIDS works in real-time, which means it tracks live data and flags issues as they happen. On the other hand, HIDS examines historical data to catch savvy hackers that use non-conventional methods that might be difficult to detect in real-time.
The ideal scenario is to incorporate both HIDS and NIDS since they complement each other. NIDS offers faster response time while HIDS can identify malicious data packets that originate from inside the enterprise network.
For the purpose of dealing with IT, there are four main types of IDS:--
Network intrusion detection system (NIDS)
Host-based intrusion detection system (HIDS)
Perimeter Intrusion Detection System (PIDS)
VM based Intrusion Detection System (VMIDS)
An Intrusion Detection System (IDS) monitors network traffic for suspicious activities and known threats, and issues alerts when such activities are discovered.
Essentially, an IDS is a packet sniffer that detects anomalies in data packets traveling along various channels. Their role is to:
Monitor systems. Assess and evaluate routers, firewalls, key management servers and files, in order to tackle cyberattacks.
Research system logs. View OS audit trails and other logs to fine-tune systems for better protection.
Identify the design of typical attacks. Match attack signature databases with information from the system.
Various components: audit data processor, knowledge base, decision engine, alarm generation and responses.
Intrusion Detection and Prevention System (IDPS) has the advantage of providing real-time corrective action in response to an attack. A passive IDS is a system that's configured to only monitor and analyze network traffic activity and alert an operator to potential vulnerabilities and attacks.
Intrusion Detection Systems (IDS) analyze network traffic for signatures that match known cyberattacks. Intrusion Prevention Systems (IPS) also analyzes packets, but can also stop the packet from being delivered based on what kind of attacks it detects — helping stop the attack.
A honeypot is a network-attached system set up as a decoy to lure cyberattackers and to detect, deflect or study hacking attempts in order to gain unauthorized access to information systems
In many cases of network intrusion, the attack involves flooding or overloading the network, gathering data about the network to attack it from a weak point later, or inserting information into the network to spread and gain access from inside. It’s important to keep hacker detection tools active, so you can prevent these vulnerabilities from getting into your system in the first place.
One kind of attack is a scanning attack, which involves sending packets or information to the network, attempting to gather data about the network topology, open or closed ports, types of permitted traffic, active hosts on a network, or what types or versions of the software are running.
Blind SQL injection attacks can use some of these scanning techniques as they attempt to find more vulnerable points in the network. Scanning attacks often look for open ports where viruses or malicious code can be inserted.
Asymmetric routing is when packets traveling through the network take one route to their destination, then a different route back. This is normal but unwanted network behavior. Depending on how your firewalls are set up, attackers can use asymmetric routing behaviors to send malicious packets through certain parts of your system to bypass your security setups.
Generally, allowing your network to perform asymmetric routing can leave you open to SYN flood attacks (a type of DDoS attack), and in most cases should be turned off for better network protection.
Buffer Overflow Attacks
This kind of attack attempts to penetrate sections of memory in devices on the network, replacing normal data in those memory locations with malicious data to will be executed later in an attack. In most cases this is also intended to turn into a form of DDoS attack.
Basically, a buffer section of memory can contain a character string or a large array of integers. A buffer overflow is when more data is written to the buffer than it can handle, which results in data overflowing into adjacent memory. This can cause the entire system to crash and can also create confusion and problems, hiding an attack in another part of the system.
These kinds of attacks target specific protocols, including ICMP, TCP, and ARP.
ICMP stands for Internet Control Message Protocol and is used by network devices to communicate with each other. The ping and traceroute tools use ICMP to communicate. One kind of ICMP attacks are also known as ping floods, in which the attacker overwhelms a device with ICMP echo-request packets.
This works because the ICMP requests require bandwidth to work, and an attack increases this network load substantially. While the device is busy dealing with this large number of malicious packets, it can’t handle normal requests.
The ICMP protocol can also be used in ICMP tunneling attacks, where data is sent through ICMP packets (which can bypass firewalls), smurf attacks, (where the address the packet is coming from is spoofed), and port scanning, where ICMP error messages are used to detect whether specified ports are open.
TCP can also be used for specific kinds of attacks, such as the TCP SYN flood. Transmission Control Protocol normally works with a “three-way handshake” to connect two devices. First, the client requests a connection by sending a synchronize (SYN) message to the server, which the server acknowledges by sending a synchronize-acknowledge (SYN-ACK) message back.
Finally, the client responds with an acknowledge (ACK) message, and the connection will be complete. A TCP SYN flood is when an attacker sends a large number of SYN messages to different ports on the targeted server, but never sends the ACK message.
As a result, the ports stay open while they wait for the ACK message to be received, during which time the attacker sends more SYN messages, the server’s connection overflow tables fill, and the system will crash or malfunction.
ARP stands for Address Resolution Protocol, and can also be used in flooding attacks, by sending large numbers of ARP packets to a recipient to overflow their ARP tables, like the above attacks. Forged or spoofed ARP packets can also be sent.
ARP poisoning is where the attacker sends false ARP messages to link the attacker’s MAC address with the IP address of a legitimate network device. Once the attacker has a valid link to this legitimate IP address, the attacker can intercept messages directed to the legitimate MAC address and modify or stop those messages.
There are several different kinds of malware, including worms, trojans, viruses, and bots. Malware is a type of software designed to damage or disrupt your system. Malware is often accidentally downloaded via email or included with another software package. Others get into your system by exploiting weaknesses in browsers, software, your network, or network devices.
Some of the most well-known types of malware are viruses and worms. These types of malware can self-replicate and spread quickly throughout your system. Worms can self-propagate, while viruses use host programs to replicate and spread.
Viruses are generally attached to .exe files, and the virus isn’t necessarily active until the user runs the malicious .exe program. As soon as the user runs this program, the virus will run as well. As they spread, viruses infect one device after another. Worms are standalone software and don’t need host programs to continue to spread. In many cases, they’ll exploit a software loophole or trick users into running them.
A trojan software disguises itself as a normal program, such as a document that looks legitimate but is malware. Unlike worms and viruses, they don’t self-replicate and can only be spread by other users.
Bots will infect a device and then connect back to a central control server. The central control server can then use all the devices infected with bots as a “botnet” to launch high-powered attacks on other targets.
In many cases, this is a type of attack called a DDoS attack, or Distributed Denial of Service attack. Some of the above attacks, including buffer overflow attacks and asymmetric routing attacks, use traffic flooding.
Intrusion detection systems are usually a part of other security systems or software, together with intended to protect information systems. IDS security works in combination with authentication and authorization access control measures, as a double line of defense against intrusion.
Firewalls and antivirus or malware software are generally set up on each individual device in a network, but as enterprises grow larger, more unknown or new devices come in and out, such as cell phones and USBs. Firewalls and anti-malware software alone is not enough to protect an entire network from attack. They act as one small part of an entire security system.
Using a fully-fledged IDS as part of your security system is vital and is intended to apply across your entire network in different ways. An IDS can capture snapshots of your entire system, and then use the intelligence gathered from pre-established patterns to determine when an attack is occurring or provide information and analysis on how an attack occurred.
Essentially, there are several components to intrusion preparation: knowledge of potential intrusions, preventing potential intrusions, being aware of active and past intrusions, and responding to the intrusion.
While it may seem “too late” once an attack has already happened, knowing what intrusions have happened or have been attempted in the past can be a vital tool in preventing future attacks. Knowing the extent of the intrusion of an attack is also important for determining your response and responsibilities to stakeholders who depend on the security of your systems.
There are three main types of intrusion detection software, or three main “parts,” depending on if you view these all as part of one system:--
Network Intrusion Detection System
Network Node Intrusion Detection System
Host Intrusion Detection System
At the most basic level, Network Intrusion Detection Systems and Network Node Intrusion Detection Systems look at network traffic, while Host Intrusion Detection Systems look at actions and files on the host devices.
What Is a Network Intrusion Detection System?
A Network Intrusion Detection System (NIDS) is generally deployed or placed at strategic points throughout the network, intended to cover those places where traffic is most likely to be vulnerable to attack. Generally, it’s applied to entire subnets, and it attempts to match any traffic passing by to a library of known attacks.
It passively looks at network traffic coming through the points on the network on which it’s deployed. They can be relatively easy to secure and can be made difficult for intruders to detect. This means an intruder may not realize their potential attack is being detected by the NIDS.
Network-based intrusion detection system software analyzes a large amount of network traffic, which means they sometimes have low specificity. This means sometimes they might miss an attack or might not detect something happening in encrypted traffic. In some cases, they might need more manual involvement from an administrator to ensure they’re configured correctly.
What Is a Network Node Intrusion Detection System?
A Network Node Intrusion Detection System (NNIDS) is like a NIDS, but it’s only applied to one host at a time, not an entire subnet.
What Is a Host Intrusion Detection System?
The Host Intrusion Detection System (HIDS) runs on all the devices in the network with access to the internet and other parts of the enterprise network. HIDS have some advantages over NIDS, due to their ability to look more closely at internal traffic, as well as working as a second line of defense against malicious packets a NIDS has failed to detect.
It looks at the entire system’s file set and compares it to its previous “snapshots” of the file set. It then looks at whether there are significant differences outside normal business use and alerts the administrator as to whether there are any missing or significantly altered files or settings. It primarily uses host-based actions such as application use and files, file access across the system, and kernel logs.
Network and host-based intrusion detection systems are the most common ways of expressing this classification, and you won’t find NNIDS mentioned very often in this space. Just think of it as a type of NIDS.
There are also two main approaches to detecting intrusion: signature-based IDS and anomaly-based IDS.
This type of IDS is focused on searching for a “signature,” patterns, or a known identity, of an intrusion or specific intrusion event. Most IDS are of this type. It needs regular updates of what signatures or identities are common at the moment to ensure its database of intruders is current. This means signature-based IDS is only as good as how up to date its database is at a given moment.
Attackers can get around signature-based IDS by frequently changing small things about how the attack takes place, so the databases cannot keep pace. In addition, it means a completely new attack type may not be picked up at all by signature-based IDS because the signature doesn’t exist in the database. Furthermore, the larger the database becomes, the higher the processing load is for the system to analyze each connection and check it against the database.
In contrast to signature-based IDS, anomaly-based IDS looks for the kinds of unknown attacks signature-based IDS finds hard to detect. Due to the rapid growth in malware and attack types, anomaly-based IDS uses machine learning approaches to compare models of trustworthy behavior with new behavior.
As a result, strange- or unusual-looking anomalies or behavior will be flagged. However, previously unknown, but legitimate, behavior can be accidentally flagged as well and depending on the response, this can cause some problems.
In addition, anomaly-based IDS assumes network behavior always stays predictable and it can be simple to tell good traffic from bad. But anomaly-based IDS looks at the behavior of traffic, not the payload, and if a network is running on a non-standard configuration, the IDS can have problems figuring out which traffic to flag.
However, anomaly-based IDS is good for determining when someone is probing or sweeping a network prior to the attack taking place. Even these sweeps or probes create signals in the network the anomaly-based IDS will pick up on. This type of IDS needs to be more distributed across the network, and the machine learning processes need to be guided and trained by an administrator.
IDS vs. Intrusion Prevention Systems vs. Firewalls
An IDS is an intrusion detection system, not a system designed to respond to an attack. An IDS can be part of a larger security tool with responses and remedies, but the IDS itself is simply a monitoring system.
Another kind of system is the Intrusion Prevention System or IPS. An IPS is essentially an IDS combined with a response or control system. IDS doesn’t alter network packets as they come through, while the IPS will prevent the packet from being delivered based on the contents of the packet (e.g., if it sees the packet is malicious).
Both a (signature-based) IDS and IPS refer to a database of known threats, and then flag or respond to the threat based on this database. IDS requires an administrator to look at the results of what has been flagged, whereas an IPS will take action automatically to block the threat.
In most ways, both IDS and IPS are largely hands-off, and even an IDS will simply send alerts to the administrator for them to respond to (the admin doesn’t have to review every single IDS process themselves). As a result, these can be useful and low-maintenance tools in your security stack.
You may also be wondering what the difference is between an IDS and a firewall. Generally, a firewall is supposed to be configured to block all traffic, and then you set it up to allow specific types through. The IDS and IPS work in the opposite direction, by allowing all traffic and then flagging or blocking specific traffic only.
As a result, you should use a firewall in combination with an IDS or IPS, not one or the other. Set up your firewall to let only specific kinds of traffic through, and then use the IDS to detect anomalies or problems in the traffic you permit. The combination of these tools provides a comprehensive security boundary for your network.
An IPS may appear more useful than an IDS simply because it “does more,” but passive listening to network behavior is still vitally important. A lot of traffic looks suspicious but is, in fact, innocent, and if your tool is always automatically blocking or reacting to traffic, you can end up with false positives interfering with normal network traffic. In either case, you need to configure your IDS or IPS to minimize false positives and negatives and to ensure accuracy as frequently as possible.
Security Information and Event Management
When you are looking into which IDS to use, you might see the term “Security Information and Event Management,” or SIEM. SIEM is a combination of Security Information Management (SIM) and Security Event Management (SEM). Basically, SEM is a kind of SIEM that looks at network traffic, like a NIDS does. These two areas overlap, and in some cases to find a good IDS tool, you’ll also need to look at tools classified as SEM software.
Choosing IDS Software
If you’re trying to decide between a host-based or network-based IDS, remember they both serve different purposes and in most cases, you’ll need both systems simultaneously, or a tool to provide both.
The host-based intrusion detection system can detect internal changes (e.g., such as a virus accidentally downloaded by an employee and spreading inside your system), while a network-based IDS will detect malicious packets as they enter your network or unusual behavior on your network such as flooding attacks or protocol-specific attacks.
Remember there are also different people in your organization who should be involved in choosing and deploying your IDS system. They include:--
Information security officers
Employees who “own” or look after sensitive data
All these people will have a good idea of network vulnerabilities and can contribute to deciding where IDS should be deployed about your network, and what kind of behavior it should be configured to detect.
If you don’t involve these critical people in choosing and setting up your IDS, you could end up missing important vulnerabilities or sensitive information in need of extra protection. The best approach is to hold a vulnerability and risk assessment meeting before you deploy and set up your IDS.
In any case, if you don’t already have an IDS set up for your business, make sure you get the process started as soon as possible. These detection systems are vital for security and shouldn’t be overlooked.
SolarWinds Security Event Manager (SEM) is an intrusion detection system designed for use on Windows Server. It can, however, log messages generated by Windows PCs and Mac OS, as well as Linux and Unix computers.
This is primarily a host-based intrusion detection system and works as a log manager. It examines connected USB devices and files coming in and out of the network and checks them for integrity and malicious modifications. It also regularly checks the filesystem itself to ensure files have not been modified, deleted, or moved in unusual ways.
SEM also can link in with and manage data collected by Snort, turning it into a more comprehensive threat-detection system with network-based intrusion detection. To use these capabilities, you need to use Snort as a packet capture tool, and then funnel this captured data to SEM. SEM can then analyze this data and detect intrusions through the network.
The tool includes knowledge-gathering processes to find relevant and up-to-date information on known botnets and other malicious actors to detect bots or intruders. This means you can configure automatic responses to known intruders, without needing to use custom scripts.
Finally, SEM can act as an intrusion prevention system as well, since it can react actively to intrusion events.
SEM can use active responses, including:---
Alerts via SNMP, screen messages, or email
USB device isolation
User account suspension or user expulsion
IP address blocking
System shutdown or restart
The multi-faceted approach of SEM makes it a comprehensive and robust tool able to take care of most incident detection, prevention, and security analysis needs.
CAPT AJIT VADAKAYIL SAYS AI MUST MEAN “AUGUMENTED INTELLIGENCE “ IN FUTURE ..
AS SOON AS BOEING 737 SUPERMAX PLANES CRASHED I WROTE THE POST BELOW.. AND I AM NOT A PILOT..
EVEN TODAY THEY HAVE NOT FIGURED OUT WHAT REALLY WENT WRONG..
The safety feature—the Maneuvering Characteristics Augmentation System (MCAS)— sent both planes into their fatal dives as pilots deprived of VETO power struggled to keep aloft
WHILE A SOPHISTICATED AI PROGRAM IS CERTAINLY CAPABLE OF MAKING A DECISION AFTER ANALYZING PATTERNS IN LARGE DATA SETS, THAT DECISION IS ONLY AS GOOD AS THE DATA THAT HUMAN BEINGS GAVE THE SYSTEM TO USE.
FULL AUTOMATION IS A MYTH.
THERE IS QUITE A BIT OF DELUSION THAT EVERYTHING CAN BE AUTOMATED. THERE ARE CERTAIN THINGS THAT CAN BE AUTOMATED, BUT HUMANS HAVE TO BE IN THE LOOP.
AT SEA WE HAVE HIGHLY AUTOMATED SHIPS.. AS A CAPTAIN, I HAD MY OWN SET OF RULES..
LATER VADAKAYILs RULES FOR AUTOMATED ENGINE ROOMS WERE ADOPTED ALL OVER SEA.
THIS INCLUDED ROUTINE CALIBRATION..
SOMETIMES CALIBRATION WAS REQUIRED BEFORE EVERY USE..
In the post above about Boeing 737 crash , although labelled an AUGMENTATION SYSTEM, MCAS is a fully automated system, invisibly compensating for redesigned characteristics of the plane -- more forward and heavier engines -- that tend to tilt up its nose.
The Maneuvering Characteristics Augmentation System (MCAS) is a flight control law (software) embedded into the Boeing 737 MAX flight control system which attempts to mimic pitching behavior similar to aircraft in the previous generation of the series, the Boeing 737 NG.
When MCAS detects that the aircraft is operating in manual flight, with flaps up, at an elevated angle of attack (AoA), it adjusts the horizontal stabilizer trim to add positive force feedback (a "nose heavy" feel), through the control column, so the pilot will not inadvertently pull the airplane up too steeply, potentially causing a stall.
Contrary to descriptions in news reports, however, Boeing claims that MCAS is not an anti-stall system— BALLS !
They escape with such bullshit because they are Jews. Stabilizer movement commanded by MCAS pushes down the nose of the airplane automatically without pilot input..
ALEXA – GIVE ME AN APPLAUSE !
WHAT BULLSHIT IS THIS?
IS THIS FUCKIN' AI ?
IS THIS FUCKIN' AI ?
Drop an item and a human child will pick it up, check for damage and move on. Unless it has been trained to undertake every one of those steps, a robot will be stymied.
No human will ever throw a diamond ring on the floor into the garbage bin, like a cleaning robot
AI agents like Cortana, can facilitate communications between people or on behalf of people, such as by transcribing a meeting and distributing a voice-searchable version to those who couldn’t attend. Such applications are inherently scalable—a single chatbot, for instance, can provide routine customer service to large numbers of people simultaneously, wherever they may be.
AI CAN BOOST OUR ANALYTIC AND DECISION-MAKING ABILITIES AND HEIGHTEN CREATIVITY
Automation is not the future, human augmentation is.. Algorithms can’t tell you how you can create value in your business, asking those sorts of questions is an innately human capability. Algorithms rely on humans having done that work first.
No matter how much AI advances humans will still be needed to oversee these systems to make sure they work properly. Machine-learning systems can already discern patterns from huge swathes of data much faster that humans can, but they are fallible.
Most of us are happy to let AI make decisions for us about trivial things, like movie recommendations, but we are less so when there’s any risk involved
Human oversight of AI apps will be needed in any regulated industry, including banking, healthcare, diagnostics and insurance, because firms will need to ensure decisions have been reached transparently and fairly, and regulators will want to know how decisions have been made.
AI should augment the way we work, allowing us to be more productive.
Of course, automation has an important role to play, especially in handling simple, repeatable processes; but it is no panacea.
Companies must understand how humans can most effectively augment machines, how machines can enhance what humans do best, and how to redesign business processes to support the partnership.
Humans need to perform three crucial roles. They must train machines to perform certain tasks; explain the outcomes of those tasks, especially when the results are counterintuitive or controversial; and sustain the responsible use of machines (by, for example, preventing robots from harming humans).
Removing humans from a context allows the automation process to redefine and adapt a given problem to solve it differently, in the interest of efficiency and reduced cost. Collaboration, on the other hand, requires adjusting to the given context and environment, and in particular to the collaborators.
Working well with people is a complex challenge, one that humanity continues to study, struggle with, and argue about. Human-machine collaboration presents promising avenues for leveraging human knowledge, skills, and qualities in ways that complement those of machines, thereby conserving human purpose.
Working with and helping people beyond physical interactions requires a much more profound understanding of people (a long-standing research pursuit) and of human-machine interaction (a recent one).
It mandates a sustained productive interaction of experts in a multitude of disciplines (cognitive, social, and health sciences, human-centered design, computing, and engineering more broadly) that as yet do not know how to converse and collaborate effectively.
Data sharing challenges and privacy concerns are only starting to fully emerge. This pursuit tests the willingness to work with highly messy, noisy, incomplete, inconsistent, and subjective human data that may not be amenable to clean mathematical models that are the current norm.
Furthermore, empathy and open-mindedness are required in accepting how little we truly understand about people. This is a rich space of research challenges and impactful applications for AI, ML, and robotics.
Machine-learning algorithms must be taught how to perform the work they’re designed to do. In that effort, huge training data sets are amassed to teach machine-translation apps to handle idiomatic expressions, medical apps to detect disease, and recommendation engines to support financial decision making. In addition, AI systems must be trained how best to interact with humans
AI assistants are now being trained to display even more complex and subtle human traits, such as sympathy. The start-up Koko, an offshoot of the MIT Media Lab, has developed technology that can help AI assistants seem to commiserate.
For instance, if a user is having a bad day, the Koko system doesn’t reply with a canned response such as “I’m sorry to hear that.” Instead it may ask for more information and then offer advice to help the person see his issues in a different light. If he were feeling stressed, for instance, Koko might recommend thinking of that tension as a positive emotion that could be channeled into action.
As AIs increasingly reach conclusions through processes that are opaque (the so-called black-box problem), they require human experts in the field to explain their behavior to nonexpert users. These “explainers” are particularly important in evidence-based industries, such as law and medicine, where a practitioner needs to understand how an AI weighed inputs into, say, a sentencing or medical recommendation.
Explainers are similarly important in helping insurers and law enforcement understand why an autonomous car took actions that led to an accident—or failed to avoid one. And explainers are becoming integral in regulated industries—indeed, in any consumer-facing industry where a machine’s output could be challenged as unfair, illegal, or just plain wrong.
For instance, the European Union’s new General Data Protection Regulation (GDPR) gives consumers the right to receive an explanation for any algorithm-based decision, such as the rate offer on a credit card or mortgage. This is one area where AI will contribute to increased employment: Experts estimate that companies will have to create about 75,000 new jobs to administer the GDPR requirements.
In addition to having people who can explain AI outcomes, companies need “sustainers”—employees who continually work to ensure that AI systems are functioning properly, safely, and responsibly.
SEB, a major Swedish bank, now uses a virtual assistant called Aida to interact with millions of customers.
Able to handle natural-language conversations, Aida has access to vast stores of data and can answer many frequently asked questions, such as how to open an account or make cross-border payments. She can also ask callers follow-up questions to solve their problems, and she’s able to analyze a caller’s tone of voice (frustrated versus appreciative, for instance) and use that information to provide better service later.
Whenever the system can’t resolve an issue—which happens in about 30% of cases—it turns the caller over to a human customer-service representative and then monitors that interaction to learn how to resolve similar problems in the future. With Aida handling basic requests, human reps can concentrate on addressing more-complex issues, especially those from unhappy callers who might require extra hand-holding.
The main objective of Aida is not to replace humans, but to be more effective at the time of dealing with customer’s queries. By doing so, human representatives have more time to focus on complex situations. Despite this, human interaction will always be needed whenever the system cannot resolve an issue
SEB,originally deployed a version of Aida internally to assist 15,000 bank employees but thereafter rolled out the chatbot to its one million customers.
Many AIs, like Aida and Cortana, exist principally as digital entities, but in other applications the intelligence is embodied in a robot that augments a human worker. With their sophisticated sensors, motors, and actuators, AI-enabled machines can now recognize people and objects and work safely alongside humans in factories, warehouses, and laboratories.
In manufacturing, for example, robots are evolving from potentially dangerous and “dumb” industrial machines into smart, context-aware “cobots.” A cobot arm might, for example, handle repetitive actions that require heavy lifting, while a person performs complementary tasks that require dexterity and human judgment, such as assembling a gear motor.
Hyundai is extending the cobot concept with exoskeletons. These wearable robotic devices, which adapt to the user and location in real time, will enable industrial workers to perform their jobs with superhuman endurance and strength.
In order to get the most value from AI, operations need to be redesigned. To do this, companies must first discover and describe an operational area that can be improved.
At Mercedes-Benz, cobot arms become an extension of the human worker’s body.
Cobots, or collaborative robots, are robots intended to interact with humans in a shared space or to work safely in close proximity. Cobots stand in contrast to traditional industrial robots which are designed to work autonomously with safety assured by isolation from human contact.
Cobot safety may rely on lightweight construction materials, rounded edges, and limits on speed or force. Safety may also require sensors and software to assure good collaborative behavior.
Thanks to sensors and other design features such as lightweight materials and rounded edges, collaborative robots (cobots) are able to interact directly and safely with humans.
The International Federation of Robotics (IFR), a global industry association of robot manufacturers and national robot associations, collects statistics on two types of robots – 1)industrial robots used in manufacturing and 2) service robots for domestic and professional use.
Service robots can be considered to be cobots as they are intended to work alongside humans. Industrial robots have traditionally worked separately from humans, behind fences, but this is changing with the emergence of industrial cobots.
Cobots can have many roles. Collaborative service robots can perform a variety of functions, from information robots in public spaces; logistics robots that transport materials within a building, to inspection robots equipped with cameras and visual processing technologies that can serve in a variety of applications such as patrolling perimeters of secure facilities.
Collaborative industrial robots can be used to automate repetitive, unergonomic tasks - such as fetching and carrying heavy parts, machine feeding and final assembly. Industrial robots have traditionally been used in industrial sectors for pre-assembly tasks such as cutting, welding, basic assembly of car bodies and painting.
Collaborative industrial robots enable automotive and electronics manufacturers to extend automation to final product assembly, finishing tasks (for example polishing and applying coatings), and quality inspection.
The IFR defines four types of collaborative manufacturing applications :--
Co-existence: Human and robot work alongside each other, but with no shared workspace.
Sequential Collaboration: Human and robot share all or part of a workspace but do not work on a part or machine at the same time.
Co-operation: Robot and human work on the same part or machine at the same time, and both are in motion.
Responsive Collaboration: The robot responds in real-time to the worker’s motion.
In most industrial applications of cobots today, the cobot and human worker share the same space but complete tasks independently or sequentially (Co-existence or Sequential Collaboration.) Co-operation or Responsive Collaboration are presently less common.
Traditionally, car manufacturing has been a rigid process with automated steps executed by “dumb” robots. To improve flexibility, Mercedes replaced some of those robots with AI-enabled cobots and redesigned its processes around human-machine collaborations.
At the company’s plant near Stuttgart, Germany, cobot arms guided by human workers pick up and place heavy parts, becoming an extension of the worker’s body. This system puts the worker in control of the build of each car, doing less manual labor and more of a “piloting” job with the robot.
The company’s human-machine teams can adapt on the fly. In the plant, the cobots can be reprogrammed easily with a tablet, allowing them to handle different tasks depending on changes in the workflow. Such agility has enabled the manufacturer to achieve unprecedented levels of customization.
Mercedes can individualize vehicle production according to the real-time choices consumers make at dealerships, changing everything from a vehicle’s dashboard components to the seat leather to the tire valve caps. As a result, no two cars rolling off the assembly line at the Stuttgart plant are the same.
For some business activities, the premium is on speed. One such operation is the detection of credit-card fraud. Companies have just seconds to determine whether they should approve a given transaction. If it’s fraudulent, they will most likely have to eat that loss. But if they deny a legitimate transaction, they lose the fee from that purchase and anger the customer.
By providing employees with tailored information and guidance, AI can help them reach better decisions. This can be especially valuable for workers in the trenches, where making the right call can have a huge impact on the bottom line.
Consider the way in which equipment maintenance is being improved with the use of “digital twins”—virtual models of physical equipment. General Electric builds such software models of its turbines and other industrial products and continually updates them with operating data streaming from the equipment.
By collecting readings from large numbers of machines in the field, GE has amassed a wealth of information on normal and aberrant performance. Its Predix application, which uses machine-learning algorithms, can now predict when a specific part in an individual machine might fail.
Tomorrow’s leaders will instead be those that embrace collaborative intelligence, transforming their operations, their markets, their industries, and—no less important—their workforces.
AUGMENTED INTELLIGENCE THE FUTURE OF ARTIFICIAL INTELLIGENCE
Human augmentation is a complementary but fundamentally different use of AI, ML, and robotics than automation
While the underlying technologies powering AI and IA are the same, the goals and applications are fundamentally different: AI aims to create systems that run without humans, whereas IA aims to create systems that make humans better.
AUGMENTED INTELLIGENCE IS A COGNITIVE TECHNOLOGY APPROACH TO AI ADOPTION THAT FOCUSES ON THE ASSISTIVE ROLE OF AI.
you augment a human beings to do their tasks a lot better or more efficiently, Here you’re not replacing anyone, but you are augmenting the skill set of that individual.
The choice of the word augmented, which means “to improve,” reinforces the role human intelligence plays when using machine learning and deep learning algorithms to discover relationships and solve problems.
Augmented intelligence emphasizes the fact that AI technologies have been developed specifically to help humans, rather than to replace them. In this way, augmented intelligence applications combine human and machine intelligence..
Augmented decision-making is the promise that if a person so chooses, then a computer can be companion, advisor and more on an ongoing, context aware, networked basis.
Augmented intelligence is an alternative conceptualization of artificial intelligence that focuses on AI's assistive role, emphasizing the fact that cognitive technology is designed to enhance human intelligence rather than replace it.
It’s a partnership between person and machine in which both parties contribute their strengths.
One of the main aims of man-computer symbiosis is to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs…
In the anticipated symbiotic partnership, men will set the goals, formulate the hypotheses, determine the criteria, and perform the evaluations. Computing machines will do the routinizable work that must be done to prepare the way for insights and decisions in technical and scientific thinking.
The future of decision-making involves a creative mix of data, analytics, and artificial intelligence (AI), with just the right dash of human judgment. The result is augmented intelligence – where the analytical power and speed of AI takes over the majority of data processing, guiding human employees to make more agile, smarter decisions and find new discoveries.
automation of processes replaces human decisions and action by technology, increasingly a combination of hardware and software. Augmentation, on the other hand, proposes that technology be used to support and improve human behavior, both in making decisions and taking action.
Think about an aircraft autopilot. In modern aviation, the autopilot can operate independently, controlling heading and altitude, or it can be coupled with a navigation system and fly pre-programed – once the aircraft has successfully become airborne.
You still need a pilot for takeoff and landing – for now. In a perfect world, an autopilot system would incorporate human knowledge, plus experience focused through the prism of intuition. The result would be human judgment extended via augmented intelligence.
The autonomous car will never be able to adapt to all situations like a human being. While a sophisticated AI program is certainly capable of making a decision after analyzing patterns in large data sets, that decision is only as good as the data that human beings gave the system to use.
Full automation is a myth. There is quite a bit of delusion that everything can be automated. There are certain things that can be automated, but humans have to be in the loop.
AI is augmentative because it solves only one part of a three-part problem – prediction first, then judgment and, finally, action
Magnetic resonance imaging (MRI) uses a variety of signals bouncing off internal body parts to predict what lies under the skin. But a doctor has to apply judgment to decide the best way to treat an injury revealed by an MRI scan
AI will augment human work while humans continue to give feedback to fine-tune the machines. No matter where the AI revolution takes us, one thing is for certain: there will always be humans in the loop.
AI has become a catch-all term, almost like “software” or “technology,” that leaves companies and society confused about what it means, its real intent and its real impact. Time and again, we see examples where current forms of autonomous AI technologies — e.g., chatbots and process automation — are not quite capable of living up to their promise of matching human intelligence
Augmented intelligence, sometimes referred to as INTELLIGENCE AUGMENTATION OR IA differs from AI as its goal is not to replace human activities, but instead to elevate existing human capabilities.
IA uses AI techniques but keeps the human in the loop. When properly applied, IA can provide feedback and insights that enhance human decision-making. Modern computing machines are still no match for the general intelligence of human beings, but IA solutions can provide tailored and timely information that helps compensate for human shortcomings and optimize their productivity.
The goal of AI should be to empower humans to be better, smarter and happier, not to create a ‘machine world’ for its own sake
The future of the augmented enterprise is clear: a blended workforce of human workers and intelligent automation technologies, in which both machines and people can do what they do best. Let machines be machines, working on repetitive, consistent, and precise processes. And let humans be human, where they can be strategic, imaginative, and engaged, doing things that only conscious humans can do,
Algorithms can’t tell you how you can create value in your business, asking those sorts of questions is an innately human capability. Algorithms rely on humans having done that work first
AI is only as good as the training data it’s given. It can never be better than the training data.
Therefore, if the training data is incomplete or lacking, expect a higher level of human in the loop
The analogy we most often like to use is that rather than the popular image of an army of robots showing up to displace people, the better analogy is knowledge workers who are fitted with AI to enable them to accomplish previously impossible tasks because of a powerful set of augmented capabilities delivered by AI
IA in many ways, is a much better bet for enterprise organizations than AI as it has a quicker ROI and better utilizes the assets already in place. Instead of targeting the lofty and extremely challenging goals of artificial general intelligence, IA can be more narrowly focused on specific high impact tasks that are achievable and concrete.
This allows employees, management, and all stakeholders to see more results in less time, enabling them to continuously make improvements and ultimately, realize a greater impact from the technology. The goal of IA is to enhance what exists, not implement a new system that is too far reaching and nebulous in its goals, and cannot justify its decision processes.
Additionally, it is important to point out that IA is not merely an intermediate stepping stone on the road to AI, where complete human replacement is inevitable. Certain functions, are part of a uniquely human skill set that will never be meaningfully realized through AI.
Emerging technologies in the Fourth Industrial Revolution will make work better, make humans more productive and create the greatest business value through its supporting role rather than replacement.
The opportunity to create the greatest business value lies in AI augmentation. But it’s up to you within your business to design the processes for augmentation to thrive.
Hybrid-augmented intelligence can be divided into two basic models: one is human-in-the-loop augmented intelligence with human-computer collaboration, and the other is cognitive computing based augmented intelligence, in which a cognitive model is embedded in the machine learning system.
More than 91% of applications that use Artificial Intelligence improve with human feedback. For example, autonomous vehicles get smarter the more that they observe human drivers; smart devices get smarter as they hear more voice commands; and search engines get smarter by observing which sites people actually click on for each search term.
Human-in-the-Loop Machine Learning Machine Learning details the process for optimizing the interaction between Machine Learning algorithms and humans who create the data that powers those algorithms
Machine Learning models can easily become biased because they are trained on data that is itself biased. Having a human in the loop can detect bias early
Because humans are needed to train most algorithms, there are now tens of thousands of people who specialize in labeling data for machine learning. People labeling data outnumber the people building machine learning algorithms and tend to come from much more diverse backgrounds
AI IS NOT A PANACEA AND IT CANNOT SIMPLY REPLACE HUMANS.
ARTIFICIAL INTELLIGENCE IS OBJECTIVE MATHEMATICAL COMPUTATION, NOT SUBJECTIVE HUMAN INTELLIGENCE.
AI may be well suited to detect digital fraud, but it would not be well suited to be a detective in the physical world. AI should be treated like any other software tool…as a product that needs to yield a return
Augmented intelligence elevates human intelligence and aids them in working faster and smarter. Augmented intelligence tools are created to help rather than replace humans.
Augmented intelligence follows a five-function cadence that allows it to learn with human influence. It repeats a cycle of understanding, interpretation, reasoning, learning, and assurance
Understanding: Systems are fed data, which it breaks down and derives meaning from.
Interpretation: New data is inputted, the system then reflects on old data to interpret new data sets.
Reasoning: The system creates “output” or “results” for new data set.
Learn: Humans give feedback on output and the system adjusts accordingly.
Assure: Security and compliance are ensured using AI technology.
Having humans and machines work hand-in-hand is a win-win for both parties. The machine grows smarter and more productive while the human workload is streamlined. With humans guiding the learning process these tools learn and adjust their models more quickly than intelligence tools with no human feedback loop.
Most often augmented tools are used to clean data sets, give predictions, improve decision-making, and to respond to customer service needs.
These systems are already in use in the healthcare, financial, retail, manufacturing, sales and marketing sectors. They’re helping diagnose and suggest treatments for ill patients in hospitals. They can perform risk analytics and regulation tasks in banks.
The beauty of augmented intelligence is that these systems use historical data to help make predictions, but the human-user always has the decision-making power.
When an organization decides to augment sales rep intelligence, productivity increases as sales reps use their time more effectively, they make smarter selling decisions and waste less time on repetitive administrative tasks that are now being automated.
Augmented intelligence impacts decisions about company spending by increasing the accuracy of sales forecasts. Sales representatives can make smarter decisions about what accounts to sell to based on their past closed deals.
Sales is the perfect opportunity for augmentation since machines can never fully replace the human element. Sales need empathy, communication, negotiation and customer service skills, in addition to being likable and relatable.
As artificial intelligence (AI) becomes prominent, it is likely to augment the digitally established human swarm, by integrating all the knowledge, data and insights that are present. These human-in-the-loop (HITL) AI, will be more powerful than AI itself due to subjective component being given the veto powers over the objective.
The synergy between humans and artificial intelligence (AI) is emerging as an effective weapon to address the current blemishes of medicine. ... Effective AI involves distinct insights in perception; in pattern recognition for text, speech, and images; in decision-making and for problem-solving.
Human-out-of-the-loop suggests that there is no human oversight over the execution of decisions. The AI system has full control without the option of human override or VETO from a conscious brain. This is most dangerous
There are tasks that require expertise and decision-making that can only be accomplished by the essential creativity that only humans could bring to the table
National security can be compromised , if an adversary feeds disinformation to a military AI system
The lack of explainability and trust hampers our ability to fully trust AI systems. We want computer systems to work as expected and produce transparent explanations and reasons for decisions they make. This is known as Explainable AI (XAI).
Traceability will enable humans to get into AI decision loops and have the ability to stop or control its tasks whenever need arises. An AI system is not only expected to perform a certain task or impose decisions but also have a model with the ability to give a transparent report of why it took specific conclusions.
AI solutions providers should build auditability and traceability into their systems. Governments and organizations using AI should consider the importance of auditability and traceability to their enforcement and compliance efforts.
To promote traceability include:--
a. Building an audit trail to document the model training and AI-augmented decision.
b. Implementing a black box recorder that captures all input
data streams. For example, a black box recorder in a selfdriving car tracks the vehicle’s position and records when and where the self-driving system takes control of the vehicle, suffers a technical problem or requests the driver to take over the control of the vehicle.
c. Ensuring that data relevant to traceability are stored appropriately to avoid degradation or alteration, and retained for durations relevant to the industry.
A black box recorder does not refer to a “black box” in the AI model sense (i.e. where the decision-making process of an AI model is inherently difficult to interpret and explain).
Reproducibility refers to the ability of an independent verification team to produce the same results using the same AI method based on the documentation made by the organisation. Reproducibility can influence the trustworthiness of the AI product and the organisation deploying the AI model.
As implementing reproducibility entails the involvement of external parties, organisations can take a risk-based approach towards identifying the subset of AI-powered features in their products or services that requires external reproducibility testing.
The evaluation of the AI system by internal or external auditors (and the availability of evaluation reports) can contribute to the trustworthiness of the AI system as it demonstrates the responsibility of design and practices and the justifiability of outcomes.
Auditability does not necessarily entail making information about business modelsor intellectual property related to the AI system publicly available.
Implementing auditability not only entails the involvement of external parties but requires disclosure of commercially sensitive information to the auditors, who may be external.
Human Centricity and Well-being: --
a. To aim for an equitable distribution of the benefits of data practices and avoid data practices that disproportionately disadvantage vulnerable groups.
b. To aim to create the greatest possible benefit from the use of data and advanced modelling techniques.
c. Engage in data practices that encourage the practice of virtues that contribute to human flourishing, human dignity and human autonomy.
d. To give weight to the considered judgements of people or communities affected by data practices and to be aligned with the values and ethical principles of the people or communities affected.
e. To make decisions that should cause no foreseeable harm to the individual, or should at least minimise such harm (in necessary circumstances, when weighed against the greater good).
f. To allow users to maintain control over the data being used, the context such data is being used in and the ability to modify that use and context.
g. To ensure that the overall well-being of the user should be central to the AI system’s functionality.
Human rights alignment: Ensure that the design, development and implementation of technologies do not infringe internationally recognised human rights.
Inclusivity: Ensure that AI is accessible to all.
Progressiveness: Favour implementations where the value created is materially better than not engaging in that project.
Responsibility, accountability and transparency:---
Incorporate downstream measures and processes for users or stakeholders to verify how and when AI technology is being applied.
To keep detailed records of design processes and decision-making.
Robustness and Security: AI systems should be safe and secure, not vulnerable to tampering or compromising the data they are trained on.
Sustainability: Favour implementations that effectively predict future behaviour and generate beneficial insights over a reasonable period of time.
Explainability and interpretability: a model is explainable when its internal behaviour can be directly understood by humans (interpretability) or when explanations (justifications) can be provided for the main factors that led to its output.
The significance of explainability is greater whenever decisions have a direct impact on customers/humans and depends on the particular context and the level of automation involved. Lack of explainability could represent a risk in the case of models developed by external third parties and then sold as ‘black box’ (opaque) packages.
Traceability and auditability: the use of traceable solutions assists in tracking all the steps, criteria and choices throughout the process, which enables the repetition of the processes resulting in the decisions made by the model and helps to ensure the auditability of the system.
Intentional bias is the most destructive for it is intended to create harm
Automated risk assessments used by U.S. judges to determine bail and sentencing limits can generate incorrect conclusions, resulting in large cumulative effects on certain groups, like longer prison sentences or higher bails imposed on people of color.
To analyse Big Data, institutions are increasingly using advanced analytics. Advanced analytics include predictive and prescriptive analytical techniques, often using AI and ML in particular, and are used to understand and recommend actions based on the analysis of high volumes of data from multiple sources, internal or external to the institution. Typical use cases include customer onboarding, fraud detection and back office process automation.
Cloud computing has been an enabler of advanced analytics, as the cloud provides a space to easily store and analyse large quantities of data in a scalable way, including through easy connectivity to mobile applications used by consumers.
Advanced analyticstechniques include those such as data/text mining, machine learning, pattern matching, forecasting, visualization, semantic analysis, sentiment analysis, network and cluster analysis, multivariate statistics, graph analysis, simulation, complex event processing, neural networks’
Data science is an interdisciplinary field involving extracting information and insights from data available in both structured and unstructured forms, similar to data mining. However, unlike data mining, data science includes all steps associated with the cleaning, preparation and analysis of the data. Data science combines a large set of methods and techniques encompassing programming, mathematics, statistics, data mining and ML. Advanced analytics is a form of data science often using ML.
Advanced analytics techniques extend beyond basic descriptive techniques and can be categorized under four headings:
Diagnostic analytics: this is a sophisticated form of backward-looking data analytics that seeks to understand not just what happened but why it happened. This technique uses advanced data analytics to identify anomalies based on descriptive analytics.
It drills into the data to discover the cause of the anomaly using inferential statistics combined with other data sources to identify hidden associations and causal relationships. The diagnostic analytics includes data mining strategy to determine the cause of a problem. It uses usability testing to determine where the user struggle through the process.
Predictive analytics: this forward-looking technique aims to support the business in predicting what could happen by analysing backward-looking data. This involves the use of advanced data mining and statistical techniques such as ML. The goal is to improve the accuracy of predicting a future event by analysing backward-looking data.
The goal of predictive analytics is to go beyond knowing what has happened to providing the best assessment of what will happen in the future. Predictive analytics uses a wide range of techniques such as data mining, statistics, modeling, and artificial intelligence to make the predictions.
Prescriptive analytics: this technique combines both backward- and forward-looking analytical techniques to suggest an optimal solution based on the data available at a given point in time.
Prescriptive analytics uses complex statistical and AI techniques to allow flexibility to model different business outcomes based on future risks and scenarios, so that the impact of the decision on the business can be optimised.
Prescriptive Analytics is the analytical technique capable of automating decision making, evaluating the best solution in complex environments. It uses the information provided by Descriptive Analytics and Predictive Analytics.
A common technique used in ML modelling is to split the available data into three groups: training data, validation data and test data. The first dataset will be used to train the model, the second dataset will be used in the next step to validate the predictive capacity of the trained model and to tune it and, finally, the third dataset will be used in the testing phase for the final evaluation of the trained, fit and tuned model.
Training data's output is available to model whereas testing data is the unseen data for which predictions have to be made..
An ML model is a mathematical model that generates predictions by finding patterns in your data.
A validation set is a set of data used to train artificial intelligence (AI) with the goal of finding and optimizing the best model to solve a given problem.. Data validation is a form of data cleansing.
Validation set is used for tuning the parameters of a model. Test set is used for performance evaluation..
Validation is an automatic computer check to ensure that the data entered is sensible and reasonable. It does not check the accuracy of data.. The training set is used to train the model, and the validation/test set is used to validate it on data it has never seen before
The training data set is a grouped set of examples that are used to fit the parameters. The test data set is the last evaluation of the final model fit on the training data set. So like the last test before it's the real deal. Testing against each other ensures the machine learning model will be more accurate.
Explainable AI will be a prerequisite for deploying any AI solution in business. Without it, AI solutions will lack transparency and trust, and prove risky. Explainable AI enables businesses to build trustworthy, ethical, and responsible AI solutions.
The lack of explainability and trust hampers our ability to fully trust AI systems. We want computer systems to work as expected and produce transparent explanations and reasons for decisions they make.
One way to achieve better model transparency is to adopt from a specific family of models that are considered explainable. Examples of these families include linear models, decision trees, rules sets, decision sets, generalized additive models and case-based reasoning methods.
Simpler models can often provide the right balance between explainability and performance (e.g., accuracy, enhanced insights). Yet, many companies dislike this approach because they believe a more complex model yields superior results.
There are two main set of techniques used to develop explainable systems; post-hoc and ante-hoc. Ante-hoc techniques entail baking explainability into a model from the beginning. Post-hoc techniques allow models to be trained normally, with explainability only being incorporated at testing time
Lack of explainability could represent an important risk in the case of AI/ML models developed by external third parties and then sold as opaque black box packages., black box products are more difficult to maintain and to integrate with other systems.
Explainability is just one element of transparency. Transparent systems could provide visibility with regard to not only the model (via explanations) but also the entire process used to build the model and the process in which the AI model is embedded when in production.
Fairness requires that the model ensures the protection of groups against (direct or indirect) discrimination
Discrimination can affect in particular smaller populations and vulnerable groups (e.g. discrimination based on age, disability, gender reassignment, marriage or civil partnership, pregnancy or maternity, race, religion or belief, sex, sexual orientation, etc.). To ensure fairness (non-discrimination), the model should be free from bias.
All the steps and choices made throughout the entire data analytics process need to be clear, transparent and traceable to enable its oversight. This includes, inter alia, model changes, data traceability and decisions made by the model.
In addition, it is important to track and document carefully the criteria followed when using the model in a way that is easily understood (e.g. including a clear indication of when a model should be retired), the alternatives (e.g. model choices) and all the relevant information on each step throughout the process.
Institutions could keep a register of the evolution of the models. Having all the versions of a model registered enables an institution to compare different models or perform a roll-back if necessary.
The steps involved in a decision made by a model can be tracked from data gathering (including from third-party data sources) to the moment when the decision is made and even beyond, as, when a model is retired, institutions could still be able to explain how its results were produced and why its decisions were made.
To enable the repetition of the process by which a decision was made, the correct version of the model and data could be used. Sometimes, the model and data will need be recovered from repositories with previous versions of models and data.
In summary, with traceable solutions, institutions are able to track all the steps, criteria and choices throughout the process, which enables the repetition of the processes resulting in the decisions made by the model and helps to ensure the auditability of the system.
Whenever there is a new technology trend, there are also new attack techniques exploiting security vulnerabilities, and AI does not escape this universal rule.
Some of the main types of attack affecting in particular ML include the following:--
-- model stealing
-- poisoning attacks
-- adversarial attacks (including evasion attacks).
Model stealing/extraction attacks are used to ‘steal’ models by replicating their internal functioning. This is done by simply probing the targeted model with a high number of prediction queries and using the response received (the prediction) to train another model. The cloned model can reach a high level of accuracy, even above 99.9%.
In poisoning attacks, attackers deliberately influence the training data to manipulate the results of a predictive model. This type of attack is especially valid if the model is exposed to the internet in online mode, i.e. the model is continuously updated by learning from new data.
An adversarial attack consists in providing a sample of input data that has been slightly perturbed to cause the model (in this case a classifier) to misclassify it. In most cases, such a small perturbation (basically representing a ‘noise’) can be so subtle that a human does not even notice it (e.g. when the problem concerns an image classification and the input image has been altered by a noise not noticeable to the human eye).
A particular example of an adversarial attack is an evasion attack, which consists in crafting the input to cause the model to avoid detection of a particular object/element.
Researchers have recently developed some new defence techniques to defeat these attacks, for example by adding a de-noiser before the inputstage. Furthermore, open-source libraries are being developed that offer techniques to test the robustness of models against this kind of attack.
Currently, this kind of attack is not yet often seen in practice, partly because, for the most part, the models implemented by institutions are not directly connected to internet, reducing exposure to such attacks.
It is important to maintain a technical watch and be regularly updated about progress on security attacks and related defence techniques.
A model is explainable when it is possible to generate explanations that allow humans to understand
(i) how a result is reached or (ii) on what grounds the result is based (similar to a justification).
Explainability requirements can be applied following a risk-based approach, becoming more stringent as the significance of the model increases (e.g. the potential impact on business continuity and/or potential harm to customers).
To support the development and functioning of models, a traceable solution (including model versioning) that assists in tracking all steps, criteria and choices through the entire process should be used. In this way, it should be possible to repeat and verify the decisions made by the model, helping to ensure the auditability and traceability of the system.
A human in the loop (where necessary) should be involved in the decisions taken by a model periodically to assess whether the model is performing correctly, depending on the criticality of the decision and the possible impact on the consumer.
To ensure that a model’s outputs remain accurate over time and that the model is not deviating from its expected behaviour, model performance could be regularly monitored (e.g. via the set-up of automatic alerts or via periodical expert reviews) and the model could be periodically updated.
The accuracy and integrity of the data need to be inspected closely to detect errors, in particular when data are from external or less trusted sources but also when using internal data. The collection of relevant and high-quality data from the beginning could result in greater accuracy than that found in data that needs extensive cleaning. However, to gain a satisfactory level of quality, data usually needs to be run through at least some cleaning procedures.
Data cleansing or scrubbing or appending is the procedure of correcting or removing inaccurate and corrupt data. This process is crucial and emphasized because wrong data can drive a business to wrong decisions, conclusions, and poor analysis, especially if the huge quantities of big data are into the picture.
There are businesses who have lost a huge amount of money due to the big bad data. Data cleansing is the procedure that filters out irrelevant data. Irrelevant data usually includes duplicate records, missing or incorrect information and poorly formatted data sets.
A business can expand this option furthermore by eliminating the data records that are not really necessary for certain business processes. While what gets filtered out depends on the discretion of the business, some basic points like outdated data or details that are not verified can be removed.
Though the data cleansing process takes a good chunk of your time as well as resources to complete, it undermines some potential of receiving major insights from the big data. Inaccurate data analytics result into misguided decision making which can expose the industry to compliance issues since many decisions are subject to requirements in order to make sure that their data is accurate and current.
Though the reduction of the potential for bad data quality can be taken care of by the process management and process architecture, it cannot be completely eliminated. The only B2B solution left is to detect and remove or correct the errors and inconsistencies in a database or data-set and make the bad data usable.
Data cleansing not only eradicates errors from both the data types but also transforms metrics and log data into a common format, providing teams with shared insights and views across the entire environment of the application. This helps the team to not only speed up their issue remediation with code and update frequency of the production code but also helps them understand the impact that their code has at any production scale and stage.
When the data is full of inaccuracies, corruptions, mixed formats, and just a mess, then your data lake qualifies as just a mud pit. Big data consists of dirty data, which requires to be cleaned to get good analytics and most probably save a lot of money.
Data cleaning, or cleansing, is the process of correcting and deleting inaccurate records from a database or table. Broadly speaking data cleaning or cleansing consists of identifying and replacing incomplete, inaccurate, irrelevant, or otherwise problematic (‘dirty’) data and records.
With effective cleansing, all data sets should be consistent and free of any errors that could be problematic during later use or analysis.
Some examples of problems that can arise out of inaccurate data are:--
Marketing: An ad campaign using low quality data and reaching out to users with irrelevant offers. This not only reduces customer satisfaction but also misses a significant sales opportunity.
Sales: A sales representative failing to contact previous customers, because of not having their complete, accurate data.
Compliance: Any online business receiving penalties from the government by not meeting data privacy rules for its customers. Facebook could be receiving such a penalty in the wake of Cambridge Analytica scandal.
Operations: Configuring robots and other production machines based on low quality operational data, can cause causes major problems for manufacturing companies
If the organization had clean data, then all of these situations (and the problems related to them) could be avoided.
Some of the biggest advantages include:--
Streamlined business practices: Imagine if there are no duplicates, errors, or inconsistencies in any of your records. How much more efficient would all of your key daily activities become?
Increased productivity: Being able to focus on key work task instead of finding the right data or having to make corrections because of incorrect data is essential. Having access to clean high quality data, with the help of effective knowledge management can be a game changer.
Faster sales cycle: Marketing decisions depend on data. Giving your marketing department the best quality data possible means better and more leads for your sales team to convert. The same concept applies to B2C relationships too!
Better decisions: We touched on this before, but it’s important enough that it’s worth repeating. Better data = better decisions.
These different benefits in conjunction generally lead to a business that is more profitable. This is not only because of better external sales efforts, but also because of more efficient internal efforts and operations.
There are a few criteria that help to qualify data as high quality. They are:--
Validity: How closely the data meets defined business rules or constraints. Some common constraints include:
Mandatory constraints: Certain columns cannot be empty
Data-type constraints: Values in a column must be of a certain data type
Range constraints: Minimum and maximum values for numbers or dates
Foreign-key constraints: A set of values in a column are defined in the column of another table containing unique values
Unique constraints: A field or fields must be unique in a dataset
Accuracy: How closely data conforms to a standard or a true value.
Completeness: How thorough or comprehensive the data and related measures are known
Consistency: The equivalency of measures across systems and subjects
Uniformity: Ensuring that the same units of measure are used in all systems
Traceability: Being able to find (and access) the source of the data
Timeliness: How quickly and recently the data has been updated
There are a few general steps that any organization can follow to start getting into a better data cleaning mindset:--
1. Develop a data quality plan. It is essential to first understand where the majority of errors occur so that the root cause can be identified and a plan built to manage it. Remember that effective data cleaning practices will have an overarching impact throughout an organization, so it is important to remain as open and communicative as possible.
A plan needs to include--
Responsibles: A C-Level executive, Chief Data Officer (CDO) if the company already appointed such an executive. Additionally, business and tech responsibles need to be assigned for different data
Metrics: Ideally, data quality should be summarizable as a single number on a 1-100 scale. While different data can have different data quality, having an overall number can help the organization measure its constant improvement.
This overall number can give more weight to data that are critical to the companies success, helping prioritize data quality initiatives that impact important data.
Actions: A clear set of actions should be identified to kick off the data quality plan. Over time, these actions will need to be updated as data quality changes and as companies priorities change.
2. Correct data at the source: If data can be fixed before it becomes an erroneous (or duplicated) entry in the system, it saves hours of time and stress down the line. For example, if your forms are overcrowded and require too many fields to be filled, you will get data quality issues from those forms. Given that businesses are constantly producing more data, it is crucial to fix data at the source.
3. Measure data accuracy: Invest in the time, tools, and research necessary to measure the accuracy of your data in real time.
4. Manage data and duplicates. If some duplicates do sneak past your new entry practices, be sure to actively detect and remove them. After removing any duplicate entries, it is important to also consider the following:--
Standardizing: Confirming that the same type of data exists in each column.
Normalizing: Ensuring that all data is recorded consistently.
Merging: When data is scattered across multiple datasets, merging is the act of combining relevant parts of those datasets to create a new file.
Aggregating: Sorting data and expressing it in a summary form.
Filtering: Narrowing down a dataset to only include the information we want
5. Append data. An append is a process that helps organizations to define and complete missing information. Reliable third party sources are often one of the best options for managing this practice.
Upon completing these 5 steps, your data will be ready to be exported and analyzed as needed. Remember that with large datasets, 100% cleanliness is next to impossible to achieve.
As is the case with many other actions, ensuring the cleanliness of big data presents its own unique set of considerations. Subsequently, there are a number of techniques that have been developed to assist in cleaning big data:--
Conversion tables: When certain data issues are already known (for example, that the names included in a dataset are written in several ways), it can be sorted by the relevant key and then lookups can be used in order to make the conversion.
Histograms: These allow for the identification of values that occur less frequently and may be invalid.
Tools: Every day major vendors are coming out with new and better tools to manage big data and the complexities that can accompany it.
About Manual Data Interventions
Today it is almost never economic to manually edit data for improvement. However, in case of extremely valuable data or when millions of labeled data points are needed as in the case of image recognition systems, manual data updates may make sense.
If manual updates will be made on the data, some best practices to keep in mind include:--
Be sure to sort data by different attributes
In the case of larger datasets, try breaking them down in down into smaller sets to increase iteration speed
Consider creating a set of utility functions such as remapping values based on CSV file or regex search-and-replace
Keep records of every cleaning operation
Sampling can be a great way to assess quality. Once you know your data quality tolerance limits, these can help you decide sample size to assess quality. For example, if you have 1,000 rows and need to make sure that a data quality problem is no more common than 5%, checking 10% of cases
Analyze summary statistics such as standard deviation or number of missing values to quickly locate the most common issues
Keeping these in mind throughout any manual data cleaning initiative can help to ensure the ongoing success of the project.
Best Practices in Data Cleaning
There are several best practices that should be kept in mind throughout any data cleaning endeavor. They are:
Consider your data in the most holistic way possible – thinking about not only who will be doing the analysis but also who will be using the results derived from it
Increased controls on database inputs can ensure that cleaner data is what ends up being used in the system
Choose software solutions that are able to highlight and potentially even resolve faulty data before it becomes problematic
In the case of large datasets, be sure to limit your sample size in order to minimize prep time and accelerate performance
Spot check throughout to prevent any errors from being replicated
Data cleaning, though essential for the ongoing success of your organization, is not without its own challenges. Some of the most common include:--
Limited knowledge about what is causing anomalies, creating difficulties in creating the right transformations
Data deletion, where a loss of information leads to incomplete data that cannot be accurately ‘filled in’
Ongoing maintenance can be expensive and time consuming
It is difficult to build a data cleansing graph to assist with the process ahead of time
Having clean data is just one part of a more comprehensive data management effort, but its importance for the best decision making possible should not be underestimated.
Data has increasingly become a critical component of just about every aspect of business and the amount of data is skyrocketing.
90% of the world’s data has been created in the last two years and it’s expected that by 2020, 463 exabytes of data will be created every day from wearables, social media networks, communications (business and consumer), transactions and connected devices.
While the explosion in the volume — and more importantly, diversity of data — is instrumental in supporting the future of Artificial Intelligence (AI) and accelerates the automation of data analysis, it’s also creating the obstacles that enterprises currently face in their adoption of AI.
Data analysis is a process of cleaning, transforming, and modeling data to discover useful information for business decision-making. The purpose of Data Analysis is to extract useful information from data and taking the decision based upon the data analysis.
Whenever we take any decision in our day-to-day life is by thinking about what happened last time or what will happen by choosing that particular decision. This is nothing but analyzing our past or future and making decisions based on it. For that, we gather memories of our past or dreams of our future. So that is nothing but data analysis. Now same thing analyst does for business purposes, is called Data Analysis.
DATA SCIENTISTS ARE SPENDING TOO MUCH TIME PREPARING DATA AND NOT ENOUGH TIME ANALYZING IT.
Data analysis is important in business to understand problems facing an organisation, and to explore data in meaningful ways. Data in itself is merely facts and figures. Data analysis organises, interprets, structures and presents the data into useful information that provides context for the data
Data analysis is the process of interpreting the meaning of the data we have collected, organized, and displayed in the form of a table, bar chart, line graph, or other representation
Data analysis has two prominent methods: qualitative research and quantitative research. Each method has their own techniques. Interviews and observations are forms of qualitative research, while experiments and surveys are quantitative research
Often data accuracy would increase if organizations were able to analyze third- party data from customers, semi-structured data, or data from relational databases.
Organizations can no longer rely on legacy, compartmentalized data integration to handle the speed, scale, and diversity of today’s data. Inadequate data cleansing and data preparation frequently allow inaccuracies to slip through the cracks.
Data collected can lose their validity over time. This is especially true for real-time data or data in high transactional environments.
Timeliness is therefore a dimension that needs to be considered when data are used in models.
There are several types of data analysis techniques that exist based on business and technology. The major types of data analysis are:---
Text Analysis is also referred to as Data Mining. It is a method to discover a pattern in large data sets using databases or data mining tools. It used to transform raw data into business information. Business Intelligence tools are present in the market which is used to take strategic business decisions. Overall it offers a way to extract and examine data and deriving patterns and finally interpretation of the data.
Statistical Analysis shows "What happen?" by using past data in the form of dashboards. Statistical Analysis includes collection, Analysis, interpretation, presentation, and modeling of data. It analyses a set of data or a sample of data. There are two categories of this type of Analysis - Descriptive Analysis and Inferential Analysis.
analyses complete data or a sample of summarized numerical data. It shows mean and deviation for continuous data whereas percentage and frequency for categorical data.
analyses sample from complete data. In this type of Analysis, you can find different conclusions from the same data by selecting different samples.
Diagnostic Analysis shows "Why did it happen?" by finding the cause from the insight found in Statistical Analysis. This Analysis is useful to identify behavior patterns of data. If a new problem arrives in your business process, then you can look into this Analysis to find similar patterns of that problem. And it may have chances to use similar prescriptions for the new problems.
Predictive Analysis shows "what is likely to happen" by using previous data. The simplest example is like if last year I bought two dresses based on my savings and if this year my salary is increasing double then I can buy four dresses. But of course it's not easy like this because you have to think about other circumstances like chances of prices of clothes is increased this year or maybe instead of dresses you want to buy a new bike, or you need to buy a house!
So here, this Analysis makes predictions about future outcomes based on current or past data. Forecasting is just an estimate. Its accuracy is based on how much detailed information you have and how much you dig in it.
Prescriptive Analysis combines the insight from all previous Analysis to determine which action to take in a current problem or decision. Most data-driven companies are utilizing Prescriptive Analysis because predictive and descriptive Analysis are not enough to improve data performance. Based on current situations and problems, they analyze the data and make decisions.
Data Analysis Process
Data Analysis Process is nothing but gathering information by using proper application or tool which allows you to explore the data and find a pattern in it. Based on that, you can take decisions, or you can get ultimate conclusions.
Data Analysis consists of the following phases:--
Data Requirement Gathering
Data Requirement Gathering
First of all, you have to think about why do you want to do this data analysis? All you need to find out the purpose or aim of doing the Analysis. You have to decide which type of data analysis you wanted to do! In this phase, you have to decide what to analyze and how to measure it, you have to understand why you are investigating and what measures you have to use to do this Analysis.
After requirement gathering, you will get a clear idea about what things you have to measure and what should be your findings. Now it's time to collect your data based on requirements. Once you collect your data, remember that the collected data must be processed or organized for Analysis. As you collected data from various sources, you must have to keep a log with a collection date and source of the data.
Now whatever data is collected may not be useful or irrelevant to your aim of Analysis, hence it should be cleaned. The data which is collected may contain duplicate records, white spaces or errors. The data should be cleaned and error free. This phase must be done before Analysis because based on data cleaning, your output of Analysis will be closer to your expected outcome.
Once the data is collected, cleaned, and processed, it is ready for Analysis. As you manipulate data, you may find you have the exact information you need, or you might need to collect more data. During this phase, you can use data analysis tools and software which will help you to understand, interpret, and derive conclusions based on the requirements.
After analyzing your data, it's finally time to interpret your results. You can choose the way to express or communicate your data analysis either you can use simply in words or maybe a table or chart. Then use the results of your data analysis process to decide your best course of action.
Data visualization is very common in your day to day life; they often appear in the form of charts and graphs. In other words, data shown graphically so that it will be easier for the human brain to understand and process it. Data visualization often used to discover unknown facts and trends. By observing relationships and comparing datasets, you can find a way to find out meaningful information.
If data is not understandable ( due to deliberate poisoning ) it can be conflicting. Without establishing context for the data ingested, the institution could find its data lake becoming a data swamp.
The ‘fit for purpose’ principle could be the basis for the time required for the establishment of a data structure for the data ingested into a data lake. The typical step is to establish only the data structures needed based on the requirements of the advanced analytics project in question. In this way, how much work is invested in describing the data ingested into the data lake is restricted.
A data lake is a storage repository that holds a vast amount of raw or refined data until it is accessed.
Some ways that AI is applied to Big Data Analytics include:-
Detecting Anomalies: AI can analyze Big Data to detect anomalies (unusual occurrences) in the data set. This can be applied to networks of sensors and parameters that have a predefined appropriate range. Any node of the network that is outside of the range is identified as a potential problem that needs attention.
Probabilities of Future Outcomes: AI can analyze Big Data using Bayes theorem. The likelihood of an event occurring can be determined using known conditions that have a certain probability of influencing the future outcome.
Recognizing Patterns: AI can analyze Big Data to look for patterns that might otherwise remain undetected by human supervision.
Data Bars and Graphs: AI can analyze Big Data to look for patterns in bars and graphs that are made from the underlying data set.
Another key driver of this trend is that Big Data is increasing through the explosion of connected devices being deployed with the expansion of the Internet of Things (IoT).
By 2025, there will be over 64 billion devices worldwide that are connected on the Internet of Things (IoT). There already are about 24 billion IoT devices.
Each device collects data. This trend is responsible for an exponential increase in Big Data. Collecting such massive data from numerous “smart” devices is only useful if it can be processed, and data mined in meaningful ways.
Text analysis tools are often used to gain valuable insights from social media comments, survey responses, and online reviews.
Text analysis allows companies to automatically extract and classify information from text, such as tweets, emails, support tickets, product reviews, and survey responses. Popular text analysis techniques include sentiment analysis, topic detection, and keyword extraction.
Businesses might want to extract specific information, like keywords, names, or company information. They may even want to categorize text with tags according to topic or viewpoint, or classify it as positive or negative.
Either way, sorting through data is a repetitive, time-consuming and expensive process if done by humans
It's very similar to the way humans learn how to differentiate between topics, objects, and emotions. Let's say we have urgent and low priority issues to deal with. We don't instinctively know the difference between them – we learn gradually by associating urgency with certain expressions.
For example, when we want to identify urgent issues, we'd look out for expressions like 'please help me ASAP!' On the other hand, when we want to identify low priority issues, we'd look out for more positive expressions like 'thanks for the help! Really appreciate it'
So, text analytics vs. text analysis: what's the difference?
One is qualitative and the other quantitative, the latter being text analytics. If a machine performs text analysis, it identifies important information within the text itself, but if it performs text analytics it reveals patterns across thousands of text data, resulting in graphs, reports, tables etc.
Text classification is the process of assigning predefined tags or categories to unstructured text. It's considered one of the most useful Natural Language Processing (NLP) techniques because it's so versatile and can organize, structure and categorize pretty much anything to deliver meaningful data and solve problems.
Emotions are essential to effective communication between humans, so if we want machines to handle texts in the same way, we need teach them how to detect emotions and classify text as positive, negative or neutral. That's where sentiment analysis comes into play. It's the automated process of understanding an opinion about a given subject from written or spoken language.
Language detection allows text to be classified according to its language and is often used for routing purposes e.g. support tickets can be automatically allocated to the right team depending on the language detected.
Text classifiers can also be used to automatically detect the intent within texts, for example, companies are able to better understand customer feedback about a product if they know more about the purpose or intentions behind the text. Anything from the intention to complain about a product to the intention to buy a product.
Text extraction is another widely used text analysis technique for getting insights from data. It involves extracting pieces of data that already exist within any given text, so if you wanted to extract important data such as keywords, prices, company names, and product specifications, you'd train an extraction model to automatically detect this information.
Then, you can organize the extracted data into spreadsheets, translate into graphs and use it to resolve particular problems. And yes, all without having to tediously sort through data, and input information manually!
Text extraction is often used alongside text classification so that businesses can categorize their data and extract information at the same time
Keywords are the most relevant terms within a text, terms that summarize the contents of text in list form. Keyword extraction can be used to index data to be searched and to generate tag clouds (a visual representation of text data).
A named entity recognition (NER) extractor finds entities, which can be people, companies or locations and exist within text data. Results are shown labeled with the corresponding entity label,
Summary extraction allows long texts to be summarized without losing their meaning. Imagine if every customer support ticket could be read in seconds instead of minutes – staff would save a lot of time, and response times would be quicker leading to better customer experiences.
Word Sense Disambiguation
It's very common for a word to have more than one meaning, which is why Word Sense Disambiguation is a major challenge of Natural Language Processing. Its function is to identify the sense of a word within a sentence when the word has more than one meaning. Easy for humans to figure out, but difficult for machines.
Take the word 'light' for example. Is the text referring to weight, color or an electrical appliance? Smart text analysis with Word Sense Disambiguation can differentiate words that have more than one meaning, but only if we teach models to do so.
Text Analysis Scope
Text analysis can stretch it's AI wings across a range of texts depending on the results you desire. It can be applied to:
Whole documents: obtains information from a complete document or paragraph e.g. the overall sentiment of a customer review.
Single sentences: obtains information from specific sentences e.g. more detailed sentiments of every sentence of a customer review.
Sub-sentences: obtains information from sub-expressions within a sentence e.g. the underlying sentiments of every opinion unit of a customer review.
Why is Text Analysis Important?
Every minute of the day, 156 million emails and 456,000 tweets are sent! That's a colossal amount of data to process, and impossible for humans to do it alone. If machines are made solely responsible for sorting through data using text analysis models, the benefits for businesses will be huge.
Text Analysis allows businesses to structure vast quantities of information, like emails, chats, social media, support tickets, documents and so on, in seconds rather than in days, and redirect extra resources to more important business tasks.
Businesses are inundated with information, making it harder to resolve urgent queries and deal with negative reviews as and when they arise! Text analysis is a game-changer when it comes to detecting urgent matters, and the big advantage is that it can work in real-time 24/7. By training these models to detect expressions and sentiments that imply negativity or urgency, businesses can automatically flag tweets, reviews, videos, tickets, and the like, and take action sooner rather than later.
Machine learning for text analysis makes it possible to process huge amounts of unstructured text data in a fast and simple way.
By creating customized models that learn from examples and improve over time, businesses can automate daily tasks and save their teams precious time, as well as gain relevant insights that enhance the decision-making process.
Big Data is a term that includes every tool and process that helps people utilize and manage huge sets of data. The concept was created out of necessity to capture trends, preferences, and user behavior into a single place (the so-called data lake) for when people interact with various systems and each other. Big Data can help companies analyze and figure out the motivations of their most important clients, while also providing ideas for the creation of new offerings.
Big Data introduces entirely new levels of uncovering hidden opportunities. Organizations couldn’t analyze such large sets of data in the past, but now the ability to do that could result in unexpected business value.
Based on data analysis, you can be alerted to problems, find new solutions, and receive ideas for new opportunities. But with the massive stream of information available, sometimes it can be really hard to determine what is important, what’s not, and how to deal with it. Machine Learning can help in the analytics process, recognize unusual patterns in processes instantly, and give you suggestions on what to do next; this process will provide valuable AI insights.
It predicts membership probabilities for each class such as the probability that given record or data point belongs to a particular class. The class with the highest probability is considered as the most likely class.
Naïve Bayes performs well in cases of categorical input variables compared to numerical variables. It is useful for making predictions and forecasting data based on historical results
In machine learning, naïve Bayes classifiers are a family of simple "probabilistic classifiers" based on applying Bayes' theorem with strong (naïve) independence assumptions between the features. a probabilistic classifier is a classifier that is able to predict, given an observation of an input, a probability distribution over a set of classes, rather than only outputting the most likely class that the observation should belong to.
Where is Naive Bayes Used?
You can use Naive Bayes for the following things:--
As a classifier, it is used to identify the faces or its other features, like nose, mouth, eyes, etc.
It can be used to predict if the weather will be good or bad.
Doctors can diagnose patients by using the information that the classifier provides. Healthcare professionals can use Naive Bayes to indicate if a patient is at high risk for certain diseases and conditions, such as heart disease, cancer, and other ailments.
With the help of a Naive Bayes classifier, Google News recognizes whether the news is political, world news, and so on.
The “artificial intelligence” moniker notwithstanding, however, these algorithms are not based on the sorts of conceptual understanding characteristic of human intelligence. Rather, they are the product of statistical pattern-matching.
Pattern recognition is the automated recognition of patterns and regularities in data. Pattern recognition is closely related to artificial intelligence and machine learning, together with applications such as data mining and knowledge discovery in databases (KDD), and is often used interchangeably with these terms.
However, these are distinguished: machine learning is one approach to pattern recognition, while other approaches include hand-crafted (not learned) rules or heuristics; and pattern recognition is one approach to artificial intelligence, while other approaches include symbolic artificial intelligence
Data mining is the process of discovering patterns in large data sets involving methods at the intersection of machine learning, statistics, and database systems. Data mining is an interdisciplinary subfield of computer science and statistics with an overall goal to extract information (with intelligent methods) from a data set and transform the information into a comprehensible structure for further use.
Data mining is the analysis step of the "knowledge discovery in databases" process or KDD. Aside from the raw analysis step, it also involves database and data management aspects, data pre-processing, model and inference considerations, interestingness metrics, complexity considerations, post-processing of discovered structures, visualization, and online updating
The difference between data analysis and data mining is that data analysis is used to test models and hypotheses on the dataset, e.g., analyzing the effectiveness of a marketing campaign, regardless of the amount of data; in contrast, data mining uses machine learning and statistical models to uncover clandestine or hidden patterns in a large volume of data
Data analysis is a process of inspecting, cleansing, transforming and modeling data with the goal of discovering useful information, informing conclusion and supporting decision-making. Data analysis has multiple facets and approaches, encompassing diverse techniques under a variety of names, and is used in different business, science, and social science domains. In today's business world, data analysis plays a role in making decisions more scientific and helping businesses operate more effectively
Data transformation is the process of converting data from one format or structure into another format or structure. It is a fundamental aspect of most data integration and data management tasks such as data wrangling, data warehousing, data integration and application integration.
Data transformation can be simple or complex based on the required changes to the data between the source (initial) data and the target (final) data. Data transformation is typically performed via a mixture of manual and automated steps.
Tools and technologies used for data transformation can vary widely based on the format, structure, complexity, and volume of the data being transformed. Data integration involves combining data residing in different sources and providing users with a unified view of them
These questions are even more troubling when you consider how central such technologies will become to all future military operations. As the technology proliferates, even morally upstanding militaries may have to rely on autonomous assistance, in spite of its many risks, just to keep ahead of their less scrupulous AI-enabled adversaries.
Once an AI system can navigate complicated circumstances more intelligently than any team of soldiers, the human will have no choice but to take its advice on trust
Israel’s Palestinians and China’s Uighur population is routinely submitted to exactly this kind of digital despotism; state and local authorities have deployed facial recognition tools capable of picking out members of the predominantly Muslim minority in closed-circuit TV footage, along with myriad other spying tools, to chart their every move.
Lethal autonomous weapons (LAWs) are a type of autonomous military robot that can independently search for and engage targets based on programmed constraints and descriptions. LAWs are also called lethal autonomous weapon systems (LAWS), lethal autonomous robots (LAR), autonomous weapon systems (AWS), robotic weapons, killer robots or slaughterbots.
LAWs may operate in the air, on land, on water, under water, or in space. The autonomy of current systems as of 2018 was restricted in the sense that a human gives the final command to attack - though there are exceptions with certain "defensive" systems.
Killer robots not guided by human remote control should be outlawed by the same type of international treaty that bans chemical weapons.
Unlike drones, which are controlled by military teams often thousands of miles away from where the flying weapon is being deployed, killer robots have the potential to do “calamitous things that they were not originally programmed for”.
Some kind of human control is necessary ... Only humans can make context-specific judgements of distinction, proportionality and precautions in combat military robots are no longer confined to the realm of science fiction or video games, but are fast progressing from graphic design boards to defence engineering laboratories.
Within a few years, they could be deployed by state militaries to the battlefield, they add, painting dystopian scenarios of swarms of drones moving through a town or city, scanning and selectively killing their targets within seconds.
Giving machines the power of life and death violates the principles of human dignity. vulnerable to interference and hacking which would result in increased civilian deaths, but their deployment would raise questions over who would be held accountable in the event of misuse.
From a military standpoint, there are various reasons why there exists huge interest in autonomous weapons.First one is Speed: Warfare has become rapidly paced. Robots as well as machines are usually faster at making quick decisions in comparison to humans, specifically when a high amount of information is required to be considered.
The second one is stealth. At the present time, a system without human intelligence like drones requires communication as well as command links such that the pilot can guide them. The issue is that such links are comparatively simple to detect. If the army has to do a sneak attack without getting noticed, autonomous weapons will do that in such a way which conventional weapons might not.
The third one why industries and armed forces are investing in autonomy and artificial intelligence is because it enables new military capabilities, especially swarms. There exists not a single way where a human can control any single drone in such kind of swarms
Russian AI-driven missiles which can decide to switch targets mid-flight. A survey by state Russian media has claimed that potential military usage of Artificial Intelligence rose in the year of 2017. Russia has even been testing numerous autonomous as well as semi-autonomous combat systems, like ‘Kalashnikov‘s “neural net” combat module’, with a camera, machine gun, and Artificial Intelligence where it can take its own targeting decisions without human intelligence.
An artificial intelligence arms race is a competition between two or more states to have its military forces equipped with the best "artificial intelligence" (AI)
“Persuasive Computing.”-- In the future, using sophisticated manipulation technologies, these platforms will be able to steer us through entire courses of action, be it for the execution of complex work processes or to generate free content for internet platforms, from which corporations earn billions.
The magic phrase is “big nudging,” which is the combination of big data with nudging.
Persuasive computing. or “captology”; the use of computing technology to change or influence attitudes or behaviors. Examples include advertising, public service messages, and demo and attract-mode screens used for point-of-sale displays and arcade games.
Persuasive technology is broadly defined as technology that is designed to change attitudes or behaviors of the users through persuasion and social influence, but not through coercion. Such technologies are regularly used in sales, diplomacy, politics, religion, military training, public health, and management, and may potentially be used in any area of human-human or human-computer interaction.
Most self-identified persuasive technology research focuses on interactive, computational technologies, including desktop computers, Internet services, video games, and mobile devices, but this incorporates and builds on the results, theories, and methods of experimental psychology, rhetoric, and human-computer interaction. The design of persuasive technologies can be seen as a particular case of design with intent
Pervasive computing, also called ubiquitous computing, is the growing trend of embedding computational capability (generally in the form of microprocessors) into everyday objects to make them effectively communicate and perform useful tasks in a way that minimizes the end user's need to interact with computers as computers. Pervasive computing devices are network-connected and constantly available.
Because pervasive computing systems are capable of collecting, processing and communicating data, they can adapt to the data's context and activity. That means, in essence, a network that can understand its surroundings and improve the human experience and quality of life.
THE INTERNET OF THINGS (IOT) HAS LARGELY EVOLVED OUT OF PERVASIVE COMPUTING.
The internet of things, or IoT, is a system of interrelated computing devices, mechanical and digital machines, objects, animals or people that are provided with unique identifiers (UIDs) and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction.
Examples include smart factories, smart home devices, medical monitoring devices, wearable fitness trackers, smart city infrastructures, and vehicular telematics.
IoT devices are often called “smart” devices because they have sensors and can conduct complex data analytics. IoT devices collect data using sensors and offer services to the user based on the analyses of that data and according to user-defined parameters.
For example, a smart refrigerator uses sensors (e.g., cameras) to inventory stored items and can alert the user when items run low based on image recognition analyses. Sophisticated IoT devices can “learn” y recognizing patterns in user preferences and historical use data. An IoT device can become “smarter” as its program adjusts to improve its prediction capability so as to enhance user experiences or utility.
IoT devices are connected to the internet: directly, through another IoT device, or both. Network connections are used for sharing information and interacting with users. The IoT creates linkages and connections between physical devices by incorporating software applications.
IoT devices can enable users to access information or control devices from anywhere using a variety of internet-connected devices. For example, a smart doorbell and lock may allow a user to see and interact with the person at the door and unlock the door from anywhere using a mobile device or computer.
Examples of pervasive computing include electronic toll systems on highways; tracking applications, such as Life360, which can track the location of the user, the speed at which they are driving and how much battery life their smartphone has; Apple Watch; Amazon Echo; smart traffic lights; and Fitbit
The idea of the Pervasive computing is embedding the computational capability into the everyday objects to make them effectively communicate and perform useful tasks in a way that minimizes the user’s need to interact with computers as computers. Unlike the desktop computers pervasive computing can occur at any time, with any device, in any place, with regardless of the data type on any given network.
So this leads IoT enabled devices to perform actions while understanding the context. The goal of pervasive computing is to make these devices smart, adapt to their surroundings and improve the human experience in day to day life.
BELOW: SOMEBODY WANTED TO HEAR MY VOICE..
I AM 105 KILOS IN THIS VIDEO..TODAY I AM 83 KGS..
I WAS 105 KILOS FOR MORE THAN 15 YEARS AT SEA-- PLENTY OF WEIGHT TO THROW AROUND..
MY JUNIOR OFFICERS BEGGED ME NOT TO LOSE WEIGHT.. THEY SAID , I AM THE BUD SPENCER AT SEA..
I AM SINGING " I AM GOING FISHING " BY CHRIS REA..
IT IS A POEM, NOT A SONG..
I'm gone fishing I got me a line Nothing I do is gonna make the difference So I'm taking the time And you ain't never gonna be happy Anyhow, anyway So I'm gone fishing And I'm going today I'm gone fishing Sounds crazy I know I know nothing about fishing But just watch me go And when my time has come I will look back and see Peace on the shoreline That could have been me You can waste a whole lifetime Trying to be What you think is expected of you But you'll never be free May as well go fishing...
TO BE HONEST, I HAVE NEVER CAUGHT A FISH IN MY LIFE .. IN MY 40 YEARS AT SEA
I WAS IN A PROFESSION WHERE AT ANCHORAGE WE CATCH FRESH FISH TO EAT
SOMETIMES ON CHEMICAL TANKERS IN THE LOADED SEA PASSAGE, CREW PLACES BRIGHT LIGHTS ON THE RAILINGS--
NEXT MORNING HUNDREDS OF DEAD BIG FISH WILL BE ON THE DECK..
BELOW: HERE I SING "ROAD TO HELL BY CHRIS REA.. IT IS ALL ON REQUEST.
WE WORKED HARD-- BUT PLAYED HARDER
THE LEGEND AT SEA SAYS-- CAPT VADAKAYILs CREW NEVER GOT BURNED OUT
Well I'm standing by the river
But the water doesn't flow
It boils with every poison you can think of
And I'm underneath the streetlight
But the light of joy I know
Scared beyond belief way down in the shadows
And the perverted fear of violence
Chokes the smile on every face
And common sense is ringing out the bell
This ain't no technological breakdown
Oh no, this is the road to hell
And all the roads jam up with credit
And there's nothing you can do
It's all just bits of paper flying away from you
Oh look out world, take a good look
What comes down here
You must learn this lesson fast and learn it well
This ain't no upwardly mobile freeway
Oh no, this is the road
Said this is the road
This is the road to hell
THIS POST IS NOW CONTINUED TO PART 18, BELOW
CAPT AJIT VADAKAYIL