~dsalaj/com/

Instead of more photos, make an audio album of your loved ones!

Wed, 14 Jan 2026 16:34:00 +0000

tl;dr: If you really want to capture a moment with someone you love, stop taking pictures and hit record on the voice memos app instead.

We are currently drowning in JPEGs. Every interesting moment results in a bunch of photos getting taken with our phones, synced to the cloud, and largely forgotten.

Don’t get me wrong, photos are great for remembering what someone or something looked like at a specific point in time. But after a while, scrolling through them feels a bit hollow. You see the face, but you don’t necessarily feel the presence.

If you want to capture your personal experience of someone you love, make candid audio recording with your phone.

Story time

🔈 I don’t know if it’s my proclivity for audio or what, but I had an intuition to start making random recordings of conversations with my grandma a couple of years before she passed away. Nothing formal, just her rambling about growing up, WW2 experiences, or even just her making random comments. I’d just set my phone on the table and let it run for twenty minutes.

Now that she’s gone, I have many photos of her. But the things I treasure most are those crappy little .ogg files.

When I look at a photo of her, I remember that she existed. When I play one of those recordings, it’s like she’s back in the room with me. The effect is… visceral. It captures the cadence of her speech, the specific way she chuckled at her own stories, the little pauses. It transports you instantly back to that moment in a way a silent image never could.

Audio is infinitely more intimate than video or photography. It forces you to focus on the personality and how the person “feels” rather than the appearance. You think you will remember exactly what your dad sounds like when he’s explaining something, or the pitch of your kid’s laugh right now, but human acoustic memory is notoriously bad. You won’t remember unless hearing it again.

So, start building an audio album. It doesn’t have to be high production value; phone microphones are amazingly good these days. Next time you’re sitting around with your parents, record a few minutes of the banter. Ask an older relative to tell you a story you’ve heard a dozens times and get it on tape. You can thank me later.

A necessary warning

Treat these recordings like sensitive documents. The technology for AI voice cloning and deep fakes is accelerating at a terrifying pace. It already only takes a minute or two of clear audio to create a passable clone of someone’s voice. For obvious reasons, you do not want your loved one’s voice data floating around in easily accessible cloud storage or public social media platforms where it could eventually be scraped and weaponized for scams.

Keep these files offline. Put them on an external hard drive, maybe even encrypt them. Keep them private. They are for you, not for the internet. Here is the text formatted with minimal Markdown for easy copying and pasting.

Similarity search and Deduplication at scale

Tue, 25 Oct 2022 23:00:00 +0000

With the increase in the application of large-scale data collection, analytics, and accompanying big data platforms over the past decade, the necessity for reliable entity matching solutions is proportionately growing. The newly available data volumes need to be integrated, processed and made usable before further value can be generated.

The fundamental problem of entity matching (also known as record linkage, data linkage, reference reconciliation, data matching, entity resolution just to name a few) lies at the core of many commercial and enterprise applications such as:

Deduplication
Similarity search
Data integration (merging of different data sources)

which cover many different domains of application like:

Retail and logistics
Ecommerce
Medical and healthcare
Advertising
Knowledge management

Given multiple collections of entity representations like tables, files or text, the entity matching problem is defined as identifying all sets of entity representations that reference the same entity. Most studies in the domain of entity matching work under the assumption that the collections of entity entries on which the algorithm is applied have homogeneous structures and similar schemas. It is also often assumed that the matching identifies only the uniquely identifiable representations, ignoring the partial matches and similarity ranks. Thus, a broader problem definition was introduced under the name of Generalized Entity Matching.

The challenges of generalized entity matching in the Big Data context cover almost all the general challenges of big data solutions including dealing with data variety (heterogeneous data sources), velocity (the algorithms must be performant enough to process the data volume and provide the results at the required time by the use-case), and budget-aware processing (even if the data volume can be processed under the time restrictions, the compute-budget is usually a hard constraint for a solution to be profitable). In this post I will focus on understanding the most important challenges and aspects of deduplication and similarity search which are critical for developing an effective solution based on these methods.

Deduplication, in broad terms, deals with data that is already available or processed into a single data source and groups the data entries into sets that represent the same entity. A natural extension, which is usually part of the deduplication solution, is a merge and resolution algorithm that can reduce the groups of semantically equal entries into a new single entry. Deduplication is usually deployed as a background ad-hoc process in an enterprise with a goal to significantly improve the data quality. It is often part of a larger restructuring process, which can be followed by other services that ensure future data quality and prevent the need for deduplication in the future. These types of solutions are either developed internally or are provided by a B2B service.

Similarity search on the other hand can be seen as a subtask within a deduplication solution, finding the same or similar entries in a data source, given a query entry or a set of attributes. The application of similarity search is broader and can often be customer-facing (B2C) through features such as document search, recommender systems, and auto-completion in data entry systems.

Challenges and constraints

When designing an entity matching-based solution, we have to take into account different considerations and nuanced relations between the methodology used and the business impact. The main question that needs to be considered is: do we need the matches between entities to have 100% confidence or not (hard matches vs soft matches)? Just imagine the logistical hell if the result of deduplication leads to the merging of two unrelated products into a single entity or the legal consequences of merging two different people into a single entry.

In practice, the different scenarios usually follow one of two patterns:

The solution is applied to the data relevant to the core business, so the matches need to be done with absolute certainty. As a consequence, we will find fewer matches because of the strict matching criteria.
→ There must be no false positives. Precision has to be 1.
The matches are not mission critical and are used in a recommender, auto-completion, search or similar system, both customer or internal facing. This allows for a more flexible definition of a match and we will find more matches overall.
→ Both precision and recall are maximized.

We will keep these two scenarios in mind when discussing the different methodologies.

Heterogeneity of data sources

The datasets or the collections of entity entries can be represented in various data formats. We can categorize those as:

Structured (tabular data, relational database)
Semi-structured (XML, JSON, graph, tagged document)
Unstructured (PDF, text, document)

When the data format is the same we can further differentiate if the collections have the same schema or not by categorizing them to:

Homogeneous
Heterogeneous

Independently of the data formats and schemas, the entries in collections can contain multiple data types:

Text
Boolean flags
Enum - generalization of boolean
List/set - generalization of many enum attributes
Binary
Graph

When developing an entity matching solution, the above properties of the specific datasets constrain the applicable methodologies we can use.

For example, if working with an unstructured dataset, we can exclude the attribute-based matching and blocking algorithms. However, if we are confident that a majority of data in the unstructured dataset reliably contains certain information, we can exploit that knowledge and extract that knowledge into attributes, and then apply the standard attribute-based matching.

Another example might be the case of working in a heterogeneous setting where we are matching semi-structured JSON data with a structured relational database. In this case, we can seek to flatten and normalize the JSON. If this is not possible we can use any of the matching and blocking methods applicable to semi-structured or unstructured datasets as we can always easily cast a more structured dataset to a less structured one.

Controlling the computational cost

During deduplication, the algorithm needs to consider all pairs of entries. It is obvious that a naive comparison of every pair of entities in the data would lead to the O(n²) run time. In the context of big data, such a solution is infeasible even on relatively small datasets. To overcome this limitation there exist many techniques such as efficient indexing, blocking or message passing techniques. These techniques reduce the search space and facilitate parallelization of entity matching algorithms and deployment on the distributed computing platforms which can significantly increase the efficiency and reduce time to solution.

Blocking is primarily used to prevent the unfeasible evaluation of the Cartesian product of all entities, but as a consequence, the reduced search space might also exclude some positive matches. Thus the question of reliability and other properties of the blocking index, such as pair completeness (recall), pair quality (precision), and reduction ratio, is important to consider within the specific problem context. Referencing back to the challenges and constraints section, let’s consider the two defined scenarios. In scenario 1, no false positives are tolerated due to implied business consequences. A lower recall and a number of false negatives can be tolerated to a degree. In scenario 2 however, a balance between precision and recall is desirable. Some minimum level of both precision and recall is needed (true positives should outnumber false positives and if possible false negatives) in order to have a useful result in the recommender system.

The most important factor to consider when choosing the blocking index is the cost-benefit trade-off. In the case of products, if we choose a high-level product category as the blocking index, the resulting blocks will be large, which leads to many unnecessary comparisons between entity pairs, which in turn leads to more computational cost and longer time to solutions. In contrast, if the blocking index is too specific, like product manufacturer and color, the resulting blocks might be too small and miss some true matches between entities that are grouped in different blocks due to bad data quality.

In an optimal case, we would already have a reliable blocking key (index) but these are mostly unavailable in heterogeneous datasets and are often unreliable with a single large homogenous data source (in the products-example above, the manufacturer attribute might have multiple different values with typos or other changes for the same manufacturer).

In cases where no existing attributes are available within entities that can be used as a blocking index, a synthetic (blocking) index can be computed. A variety of algorithms are available for computing the synthetic blocking index including Bigram Indexing, Sorted Neighborhood, and Canopy Clustering. The same cost-benefit trade-off applies to the synthetic blocking indexes, which are even more prone to sensitivity loss (losing the positive matches) when applied as a black box solution.

For more critical solutions, one often includes some domain knowledge about the data and attributes when computing the synthetic blocking index. For example, one might exploit unreliable partitioning keys (like product manufacturer) to increase the coverage confidence (by clustering manufacturer values and using cluster indices) while still significantly reducing the search space.

When dealing with data that lack useful metadata or attributes for blocking, one might have to compute the blocking index based on the content of entities. For this case, there is a possibility of using (deep learning) embeddings of data entries as the input to the clustering algorithm for producing a more reliable blocking index or finding the n-nearest-neighbors for comparison.

For even more demanding use cases like continuous entity matching, please refer to these two comprehensive reviews of all available state-of-the-art blocking methods.

After having the search space significantly reduced with blocking, it is still critical to exploit the possible parallelization of the matching process, especially in the case of deduplication or the partial pre-computation of similarities for similarity search. One consequence of parallel matching is that the result can contain overlapping groups, so an additional merging step needs to be introduced.

Further optimizations can be performed on the level of query picking (sampling strategy) during the iterative deduplication process, but are costly and highly dataset-specific.

Identifier based matching

An identifier within a data source is an attribute which already matches identical entities. Every identifier is only valid on a certain scope, which typically does not cover all the available data (otherwise the problem would be already solved). It is often the case that during entity matching one has to exploit available identifiers which are incomplete, partially available, or span different scopes of the data. For example, if the dataset consists of multiple data sources, each data source can have its own unique identifiers. Some identifiers can have scopes that cover multiple data sources but only partially.

To exploit such overlapping partially-available identifiers, we can employ an iterative matching method to extend the scope of entity matching to the whole dataset. The main advantage of using available identifier is the 100% confidence in the matches as required for scenario 1 in the “Challenges and constraints” above (entity matching impacting the core business processes).

Partially-available identifiers can also be used as labels for training the classifiers of attribute-based or deep learning-based matching approaches.

Another very impactful methodology is using the identifiers (or attributes) as exclusion criteria. Those can be used in conjunction with matching criteria in the defined order of precedence to decide if the pair of representations is a true match. One can also take it a step further and integrate the exclusion criteria in the calculation of blocking index, where the search space can be reduced even further.

Attribute-based matching

Entities in structured datasets have a multitude of attributes, whose relevance for entity matching is not always clear. The similarity between individual attributes of different entities can be computed but takes on a different form depending on the data type. The following figure shows examples of the data types, their features and similarity metrics that can be computed on them.

Data type	Feature	Similarity / Distance metric
Boolean	Boolean	Equality
Int	Int	Equality
Float	Float	Difference
Enum	Enum	Equality
Set (of enums or strings)	Set	Overlapping fraction
Geo-Coordinates	Geo-Coordinates	Euclidean distance Tunnel distance Ellipsoidal-surface distance Road distance
String	Normalized String	Smith-Waterman Edit distance Q-gram distance Jaro-Winkler distance Monge-Elkan distance Extended Jaccard coefficient SoftTFIDF similarity Longest Common Substring similarity Bag distance Compression distance Editex similarity Syllable Alignment distance
String	Vector Embedding	Euclidean distance Cosine similarity Dot product

After the similarity scores of all attributes between the two entities are computed, they need to be combined into a single similarity metric. This can be done in different ways, from domain knowledge-driven manual weighing of individual similarity scores to automated data-based methods that require a labeled dataset. For the later variant, any type of classifier model can be trained. Depending on the properties of the available dataset, one usually has to apply one of the following two approaches:

Generating the dataset with manual labeling and active learning
Exploiting the available unique identifiers as labels

Unique identifiers that are valid on a substantial subset of data are optimal to reuse for generating a labeled data set. If the scope of the unique identifier is too small, then the labeled dataset it produces might be too biased and not directly useful for training. This however can be used as a starting point for the active learning approach which is still better than starting from scratch.

If no partial identifiers are present that can be used as a labeled dataset, then the only available alternatives are the manual weighing of the similarity matrices. This is bound to produce suboptimal results but might be used as a baseline in cases there are no alternatives. The baseline can then be used as a starting point for active learning. Active learning methodology supports the manual labeling process to maximize the performance gains for every labeled sample (by picking the samples for labeling based on the highest uncertainty of the classifier model).

Independently of the dataset quality and the classifier confidence, with attribute matching, we can never be 100% confident in the matches as required for scenario 1 in the introduction. The only way to really use the attribute-based matching for scenario 1 is to have a manual validation step after the match generation. However, the uncertainty of the attribute-based matching is not an issue for scenario 2 where the output of the matching is only used as a recommendation generator in a business process or a customer experience.

Deep learning approaches

Despite the ongoing research since 1946 in different aspects and domains of entity matching, the problem is still open and does not have a satisfactory solution. The requirements of intensive human involvement in feature engineering, tuning, manual labeling, and integrating domain knowledge into the entity matching solution largely remain. With the advances in deep learning, novel methods enable approaching the entity matching problem from a different perspective and potentially increasing the performance and reducing the need for human involvement in the development process.

Promising deep learning concepts that enable a different type of entity matching are word and document encoders. These neural networks are trained to produce embeddings in latent space whose distance should be proportional to the similarity of the entities in the target domain. Such neural-network-based encoders come in different varieties and are suited for several domains. For example, using the distributed word representations with RNNs and LSTMs, the authors of DeepER were able to develop a novel entity matching system with high accuracy and efficiency that requires less human effort. An overview of deep learning-based entity matching approaches can be found here.

While the above methods improved multiple aspects of entity matching and set the new state-of-the-art benchmark on datasets, they still rely on having available large and good-quality labeled datasets which are mostly not available in a real-world setting. Thus, novel deep learning approaches have been developed to specifically tackle the low resource settings. For example, an architecture combining transfer learning and active learning, dubbed Deep Transfer Active Learning, was developed and exploits the publicly available datasets to train a high-resource model and transfer-learn to a low-resource target setting with very efficient active learning labeling, minimizing the human effort while still retaining the high performance. Another innovative deep learning approach is the Ditto architecture which enables direct entity matching on datasets with heterogeneous schemas.

Methods that can completely overcome the need for labeled datasets are self-supervised learning approaches, like CollaborER and KRISS, which achieve state-of-the-art entity matching performance and can even outperform some supervised-learning-based methods.

In the follow-up post, I will cover more advanced techniques and architectures in entity matching including federated entity resolution and real-time data entity matching.

Pruning and Sparsification of Neural Networks

Sat, 14 Aug 2021 23:00:00 +0000

The pruning practice entails targeted removal of diseased, damaged, dead, non-productive, structurally unsound, or otherwise unwanted tissue from crop and landscape plants. [Wikipedia]

Just as the practice of pruning helps shape and direct the growth of trees and bushes, so does pruning of synaptic connections in the brain play a critical role in learning. The use of pruning in neural network models is gaining popularity due to the many benefits it offers.

Artificial neural networks (ANNs) are traditionally dense and fit the current computing paradigm and architecture of GPUs well. This match between the available hardware and ANN architectures was the major factor that led to the exciting developments in AI within the last decade.

In recent years, novel ANN models have been increasing in size and complexity to such an extent that the memory and computation costs have become a major issue and the focus of the research community [Strubell et al. 2020]. Another consequence is that large models are over-parametrized, which might cause issues with generalization, as the models get too sensitive to subtle input changes [Kohler and Krzyzak 2019, Hoefler et al. 2021]. These issues are usually worked around by using various methods during the training phase, such as:

L1 or L2 weight regularization
Dropout regularization variants
Data Augmentation

However, none of these methods deals with the fundamental issues of over-parametrization: inefficient use of memory and computation.

Pruning or sparsification is a family of methods to produce sparse neural networks. Similar to other regularization methods, pruning often leads to a better generalization of the networks [Bartoldson et al. 2020]. Many recent works have also shown that sparse models tend to be more robust against adversarial attacks [Cosentino et al. 2019; Gopalakrishnanet al.2018; Guo et al.2018; Madaan et al.2020; Rakin et al.2020; Sehwag et al.2020; Verdenius et al.2020]. Sparse models have also significantly lower memory and computation costs compared to dense counterparts (often in the range of 10-100x reduction). This is especially relevant for the training phase because the large and dense models are not only very expensive during inference, but often are prohibitively costly to train [Brown et al. 2020It is important to keep in mind that computation and memory efficiency depend largely on the implementation of the sparse data types and operations on them. Different hardware and different libraries offer a variety of features and support for sparse network models.]. Thus, the pruning methods that are applied during training are the most interesting and beneficial for reducing computational costs.

Typical test error vs. sparsity showing Occam’s hill (results: ResNet-50 on Top-1 ImageNet). Reproduction of Fig. 4 from [Hoefler et al. 2021].

It is important to keep in mind that computation and memory efficiency depend largely on the implementation of the sparse data types and operations on them. Different hardware and different libraries offer a variety of features and support for sparse network models.

Types of sparsity

Different pruning methods can be characterized by three main aspects:

which parts of the model are pruned
when is the pruning applied
how is the pruning implemented

The sparsification or pruning can be permanently applied to the model, also called structural pruning, or temporarily during computation, also called ephemeral sparsification.

Structural pruning in its basic form relates to the fine-grained pruning of weights of a neural network, but can also be applied over larger building blocks of models such as neurons, convolutional filters, and attention heads. Neural architecture search (NAS) is a family of such block pruning methods that aim to automate the search for better deep learning architectures [Elsken et al. 2019]. The advantage of coarser element (block) style pruning is that the block and the architecture of the model stay dense and efficiently executable on common hardware. In contrast, fine-grained pruning of weights can result in unstructured layers which can result in computation and memory overhead depending on the specific architecture and data types used. Fine-grained pruning, however, is much more flexible and when paired with specialized hardware and sparse activations (like neuromorphic chips and spiking neural networks) can result in a much stronger reduction in computation and memory requirements.

Ephemeral sparsification refers to the sparsification happening online, usually dependent on the current input data samples. This type of sparsification is very common in deep learning models and include sparse activations like ReLU, which clamp the output of a neuron to zero for input values below zero, and regularization methods such as Dropout which effectively clamp the output of a random sample of neurons to zero during training. Less common methods that belong to this category include sparsification of gradients during training and conditional computation.

Overview of neural network elements that can be pruned. Reproduction of Fig. 6 from [Hoefler et al. 2021].

When to prune?

Pruning can also be applied at different points during the training of the neural networks. The most trivial and the least beneficial application of pruning is after the training. In this setting, the model is sparsified after the training, which usually results in significant performance degradation. It also means that the full dense model is trained, which is the most costly process in terms of memory and computation.

A better approach is using a schedule of sparsification during training. This way training is started with a dense model which is gradually sparsified during the training procedure. Different pruning schedules and implementation details [Wortsmanet al. 2019; Lin et al. 2020] make this method very flexible. The advantage of this method is that sparsification can prevent overfitting gradually during training. But it can also make convergence more unstable.

[Prechelt 1997] used the generalization loss to quantify the amount of overfitting and adjust the pruning rate dynamically during training. [Jin et al. 2016] combined the dense and sparse training in a single schedule with a method called iterative hard thresholding (IHT). [Han et al. 2017] used a three-step schedule (dense training to convergence, magnitude-pruning followed by retraining, and dense training) and showed that this schedule leads to significantly higher generalization performance. A follow-up extension to this method is presented in [Carreira-Perpinan and Idelbayev 2018]. A major downside to all these approaches is that they still start with dense models which need to be held in memory and thus do not enable the use of smaller-memory devices.

The last category is a fully-sparse training, where training starts with an already sparse model and trains it in the sparse regime by adding and removing weights during training. These methods often have more hyperparameters, but enable very high-dimensional models to be trained on smaller memory hardware. There is a variety of pruning and regrowth techniques that can be combined to implement a fully-sparse training scheme. For example, [Mostafa and Wang 2019] use random regrowth and magnitude pruning to maintain sparsity throughout training.

Overview of structural sparsification schedules. Reproduction of Fig. 7 from [Hoefler et al. 2021].

State of the art

Using sparse evolutionary training approach [Mocanu et al., 2018] demonstrates training of sparsely initialized networks by dynamically changing the connectivity with a magnitude-based pruning and random growth strategy. A related method [Bellec et al., 2018] demonstrates the advantage of training sparse networks with stochastic parameter updates, by sampling the sparse connectivity pattern based on a posterior which is theoretically shown to converge to a stationary distribution.

Authors of [Mostafa & Wang, 2019] propose dynamic sparse parameterization to train sparse networks via adaptive threshold-based pruning. In [Dettmers & Zettlemoyer, 2019] authors develop a sparse learning method using momentum-based regrowth of pruned weights.

More recently, with the rising concerns about the adversarial attacks, [Özdenizci and Legenstein 2021] propose an intrinsically sparse rewiring approach based on Bayesian connectivity sampling to train neural networks by simultaneously optimizing the sparse connectivity structure and the robustness-accuracy trade-off.

Summary

In conclusion, the pruning and sparsification of neural networks can provide the following benefits:

improved generalization
robustness to adversarial attacks
memory and compute efficiency

Many popular deep learning libraries already provide many built-in pruning methods:

A large part of the content of this article is based on [Hoefler et al. 2021] and [Özdenizci and Legenstein 2021]. The figures are reproduced with written permission from Prof. Hoefler.

All about Positional Encoding

Tue, 02 Mar 2021 23:00:00 +0000

What is positional encoding?

Positional encoding was originally mentioned as a part of the Transformer architecture in the landmark paper “Attention is all you need” [Vaswani et al., 2017]. This concept is first introduced under the name of position embedding in [Gehring et al., 2017] where it was used in the context of sequence modelling with convolutional architectures.

In the Transformer architecture, positional encoding is used to give the order context to the non-recurrent architecture of multi-head attention. Let’s unpack that sentence a bit. When the recurrent networks are fed with sequence inputs, the sequential order (ordering of time-steps) is implicitly defined by the input. However, the Multi-Head Attention layer in the Transformer is a feed-forward layer and reads a whole sequence at once. As the attention is computed on each datapoint (time-step) independently, the context of ordering between datapoints is lost and the attention is invariant to the sequence order. The same is generally true for other non-recurrent architectures like convolutional layers where only a small sequential ordering context is present, limited by the size of the convolution kernel.

To alleviate this problem, the concept of positional encoding is used. This is nothing more than adding a tensor (of the same shape as the input sequence) with specific properties to the input sequence. The positional encoding tensor should be such that the value difference of the specific steps in the sequence correlates to the distance of individual steps in time (sequence).

What is Absolute Positional Encoding?

The simplest example of positional encoding is an ordered list of values, between 0 and 1, of a length equal to the input sequence length, which is then tiled to the same number of features as the network input and added to that input.

Authors of [Vaswani et al., 2017] however proposed a different absolute positional encoding based on the sine and cosine functions:

where pos is the position in time, dmodel is the number of dimensions and i is the dimension index in the input tensor. This way each wavelength is a geometric progression from 2π to 2kπ. To make this more clear here is a code snippet that plots this positional encoding table for an imaginary input tensor with 200 time steps and 12 channels:

import numpy as np
import seaborn as sns

def sinusoid_positional_encoding_ref(length, dimensions):  
    def get_position_angle_vec(position):
        return [position / np.power(10000, 2 * (i // 2) / dimensions) for i in range(dimensions)]

    PE = np.array([get_position_angle_vec(i) for i in range(length)])
    PE[:, 0::2] = np.sin(PE[:, 0::2])  # dim 2i
    PE[:, 1::2] = np.cos(PE[:, 1::2])  # dim 2i+1
    return PE

pe = sinusoid_positional_encoding_ref(200, 12)

sns.set(rc={'figure.figsize':(14,4)})
ax = sns.heatmap(pe.T)
ax.invert_yaxis()
ax.set_ylabel("input dimension")
ax.set_xlabel("time step")
ax.set_title("Sinusoid absolute positional encoding")

The authors hypothesize that this type of absolute positional encoding supports models in learning to attend to relative positions, as any fixed offset k, PEpos+k is representable as a linear function of PEpos.

But why not just use the one-hot encoding of position? Ideally, we want the attention mechanism to be able to generalize to the input sequences longer than the ones encountered in the training set. Using the one-hot position encoding, even if the dataset contained a uniform distribution of all input lengths that the model need to generalize to, the model would still be biased as it would learn the embedding of earlier positions much better because it would encounter them more often during training.

If we think about this problem, we may conclude that the same bias might occur even with the sinusoid-based absolute positional encoding. This led to a follow-up study [Shaw et al., 2018] where the issue of generalizing to arbitrary input lengths is tackled, and the relative positional encoding is introduced.

Another variant of absolute positional encoding exists where the position embeddings are learned jointly with the network model during training. This method suffered from the same generalization issues as the fixed absolute positional encoding methods.

What is Relative Positional Encoding?

The paper “Self-Attention with Relative Position Representations” [Shaw et al., 2018] introduces the concept of relative positional encoding. This concept incorporates the explicit representations of relative position in the self-attention mechanism which significantly improves the performance of the Transformer architecture.

Conceptually the relative positional encoding allows the self-attention to consider arbitrary relations between any two elements in the input. The authors of [Shaw et al., 2018] focused on the efficient implementation, and the change to the self-attention equations is minimal:

$$\text{Self-Attention}\\ z_i = \sum_{j=1}^{n}\alpha_{ij}(x_jW^V)\\ \quad \\ \alpha_{ij}=\frac{\exp e_{ij}}{\sum_{k=1}^n\exp e_{ij}}\\ \quad \\ e_{ij}=\frac{x_iW^Q(x_jW^K)^T}{\sqrt{d_z}}$$ $$\text{Relation-aware Self-Attention}\\ z_i = \sum_{j=1}^{n}\alpha_{ij}(x_jW^V + a_{ij}^V)\\ \quad \\ \alpha_{ij}=\frac{\exp e_{ij}}{\sum_{k=1}^n\exp e_{ij}}\\ \quad \\ e_{ij}=\frac{x_iW^Q(x_jW^K + a_{ij}^K)^T}{\sqrt{d_z}}$$

As we can see the only change is in the addition of the new vector to the products of key and value matrices with the input. This new vector is trained jointly with the model and is supposed to learn the additive component which is going to augment the previously mentioned product xW with the relative position context. To make it clearer let’s look at how this vector is built.

\[a_{ij}^K=w^K_{\text{clip}(j-i, k)}\\ a_{ij}^V=w^V_{\text{clip}(j-i, k)}\\ \text{clip}(x,k) = \max(-k, \min(k, x))\]

So the relative distance context this vector can represent is clipped to k in both directions. For example, if we have k=3, we learn only the relative position embeddings up to distance 3 between the tokens. When the distance between tokens is greater than 3, the embedding returns the same value as for distance equal to 3. Here is a visualisation of such an imaginary learned positional vector:

The main advantage of this method is that the positional embedding depends only on the relative distance between the tokens, and not on their absolute position in the sequence. This should lead to a more robust generalization of the attention beyond the sequence lengths present in the training set. This has been demonstrated in [Shaw et al., 2018] by achieving significantly better results than the models with absolute positional encoding. More methodical comparison of the two methods has been conducted in [Rosendahl et al., 2019] where it is again shown that the relative positional encoding is superior when translating sequences that are longer than any observed training sequence

What is RNN based positional encoding?

Since recurrent layers have implicit information about the order of input sequence, these have often been used as a layer preceding the Transformer or any other attention mechanism to endow the input sequence with the order context. This type of RNN+Attention architecture was usually trained jointly which often leads to the result where RNN is able to perform a lot of the computation required for solving the task and not only give the positional embedding to the input.

Transformers trained directly on the input sequences allow for easier interpretation, as the attention layers can be probed to reveal the learned relationships between elements in the input sequence. When an RNN is placed in front of the Transformer, the input gets convoluted which makes such interpretations impossible.

Positional encoding for tree-structured data

So far we discussed positional encoding use for processing sequential data with attention based architecture. However there is a growing interest in applying the attention methods on the tree-structured data and tasks like programming language analysis, language parse trees translation, and semantic parsing like extracting a database query from a natural language request.

The paper [Shiv and Quirk, 2019] introduces tree positional encoding using a stack-like encoding for tree positions. The authors demonstrate a significant increase in the performance of transformers using tree positional encodings for a multitude of tasks compared to the original sequence focused transformers.

How to do positional encoding for irregularly sampled data?

There are quite a few options for dealing with irregularly sampled data, or time-series data with non-equidistant timesteps. I discuss this in more details in a different post:

Corrupt, sparse, irregular and ugly: Deep learning on time series

How to migrate binary bloat out of git repository

Sat, 24 Oct 2020 23:00:00 +0000

The situation: A project has several binaries that need to be tracked and versioned along with the code. The binaries are small enough and rarely change, so versioning them directly in git is appropriate.

That was some months ago, and in the meantime, the project evolved, binaries grew and changed. Now you're left with a bloated repository of multiple gigabytes that takes a long time to clone. This is significantly slowing down the CI pipeline and it's time to fix it.

Dealing with symptoms directly

There are many options to solve this problem, all with unique pros and cons. The simplest solution is dealing only with problematic symptoms like slow CI pipeline. This can be overcome with one of the following:

Shallow clone git clone --depth 1 URL: Limiting the depth of the clone means that the history is not pulled.

Clone without binary files git clone --filter=blob:none URL: The filter option is more powerful and can be used filter only specific files that meet specific criteria. See docs.

Finally, if the large binary file size is not your problem but the large number of files, you should look into git sparse-checkout. See docs.

But sometimes these workarounds are not enough and you need to rewrite the repo history. Below I will discuss the two solutions that should cover most of the usecases. The first option is migrating to the git lfs. The second option is pruning the binaries from the repo history using git filter-repo and versioning the binaries in a separate versioning system.

Migrating to `git lfs`

The main advantage of the migration to git lfs is that the repo history is left intact and no information is lost. This method is recommended when the binaries are really tightly coupled with the code. This is easily done with the following steps:

Install the git-lfs using this guide or in conda like so: conda install -c conda-forge git-lfs

In your repo, run: git lfs install

Rewrite history. For example migrating all bz2 files: git lfs migrate import --include-ref=master --include="*.bz2"

For more details on how to match your usecase see this guide.

The downside of using git lfs is that it adds more complexity to the workflow of developers. Decisions need to be made: Which files are to be versioned with git and which with git lfs? Do you decide based on the file size or the file format/extension? The .gitattributes file needs to be maintained etc.

You get the gist, there is overhead and nobody likes overhead. But if the binary files are really tightly coupled with the code and need to be version, this is the way to go.

Pruning and migrating out of git

When the binaries are not tightly coupled with the code there is another solution: simply prune away all the binary files from the repo history and version them in a separate, more appropriate system.

The pruning part is simple thanks to the amazing git filter-repo tool. For example, pruning away all the binary files larger than 10 MB can be done using the guide from GitLab. For more elaborate cases please refer to the official documentation with examples.

After the pruning, your repo should be very lean, and you are free to version or track the binaries outside of git. For example, in the project I was working on the mlflow was the appropriate tool we decided on after some trial period. Using the mlflow Python API it was easy to track any artifacts and results along with the meta-data which matched the needs of the project well.

The obvious downside is that you lose the history of the binary files. This was the trade-off we were willing to take as the history of the binary files was not very valuable, and we still had access to it in the backups of the repo we made before the prune.

Hope this was helpful and you were able to get some useful pointers and links that will help with your git endeavors. Let me know if you found a different solution. Cheers! 🍻

Corrupt, sparse, irregular and ugly: Deep learning on time series

Sat, 20 Jun 2020 23:00:00 +0000

Update 24.06.2020: Thanks to BadInformatics for pointing me to ODE based models and other remarks. The content was extended accordingly.
Update 22.08.2020: Extended the details on missing values; Replaced the first figure with example and code; Added tip for Traces library.
Update 01.11.2021: Added Multi-Time Attention Nets and Neural Rough Differential Equations method

How to train neural networks on time-series that are non-uniformly sampled / irregularly sampled / have non-equidistant timesteps / have missing or corrupt values? In the following post, I try to summarize and point to effective methods for dealing with such data.

What is irregularly sampled data or sequence of non-equidistant steps? 📊

Time series where the time between the individual steps/measurements is not constant is called non-uniform or irregularly sampled. Irregularly sampled data occurs in many fields:

Meteorological data
Astronomical data
Electronic health records (EHRs)
Turbulence data
Automotive data

This is also common in cases where the data is multi-modal. Multi-modal input means that the input is coming from multiple different sources which most likely operate and take measurements not synchronized with each other resulting in the non-uniform input data.

Another source of the irregularity in sampling can be missing or corrupt values. This is very common in both automotive and EHR data where sensors malfunction or procedures are interrupted or rescheduled.

Why not just interpolate? 💡

The first idea you might have is to interpolate and resample the time series data at hand. This will produce a uniformly sampled grid and allows for standard methods to be applied. However, this approach works under the strong assumption that the interpolated data behaves monotonically between the measurements. This assumption often does not hold which leads to undesired artifacts during feature transformation and in turn to a sub-par performance (sometimes worse than training on the raw data).

💡 If you are looking for help in transforming unevenly-spaced times series to evenly-spaced ones, check out the Traces python library.

Another factor to consider before interpolating is the distribution of the sizes of timesteps. The easiest way to do this is to plot the histogram of dt-s (time differences) between data points. If the values of dt are relatively large, it is unlikely that the measurements in between the points behave monotonically. Furthermore, you should check the variance of dt and decide whether it is “sufficiently” uniform to apply the classical methods or not, as sometimes it turns out that the data is “uniform enough” (see the next section).

An example of dt distribution of a real world dataset (Gaia European space mission Data Release 2) with irregular time series is shown in the plot below. The plot is generated using this python notebook.

Another factor to consider is the computational and memory cost of interpolation which depends on both the size of your dataset (if you plan to do it offline) and the targeting granularity of the interpolation grid.

However, if you have a sufficiently large amount of data where you could learn the underlying distributions, then it might be worth looking into the methods for data recovery and super-resolution which I describe in the section below.

Solving the problem by ignoring the problem 🙈

There are cases when the variance in the dt-s is small enough and the methods for regularly sampled data can produce surprisingly good results. I would highly suggest first creating a baseline with one such method. For example, you can use any STFT based spectrogram (like MFCC) or any other appropriate spectrogram and train the model using such input.

Having a strong baseline is critical for evaluation of the more advanced input processing methods and the decision if they make sense for your specific setup.

What are the methods for working with irregular or sparse time series? ⚗️

If the distribution of dt is wide in your dataset, then there are several methods for directly dealing with irregularly/nonuniformly sampled data that are worth looking into:

Lomb-Scargle Periodogram
Data modeling with Interpolation networks
Neural Ordinary Differential Equation models ✨
Add timing dt to the input as an additional feature (or positional encoding)
Methods for dealing with missing values

How to use Lomb-Scargle Periodograms?

tl;dr Here is a python notebook that shows how to compute a spectrogram of irregularly sampled data using Lomb-Scargle method.

If one wants an actual spectral analysis of non-uniformly sampled signals, then Lomb-Scargle periodogram is a classical method solving the problem. The method is similar to the Fourier Power Spectral Density (PSD), the most popular method for periodicity detection in uniformly/regularly sampled data. Lomb-Scargle Periodogram algorithm has been implemented in many libraries (scipy, gatsby, astropy, matlab), out of which the scipy version should be the most maintained one.

The Lomb-Scargle periodogram was developed by Lomb [Lomb, N.R., 1976] and further extended by Scargle [Scargle, J.D., 1982] to find, and test the significance of weak periodic signals with uneven temporal sampling. [scipy docs]

A periodogram is an estimate of the spectral density of a signal. These libraries provide a method to compute a periodogram (spectral analysis) on a sequence of irregularly sampled values. However, for deep learning we usually need the spectrogram and not a single periodogram.

A spectrogram is a representation of the spectral density of frequencies of a signal over time. Thus, we can produce a spectrogram by computing the periodogram over time bins of a signal or time series. Example code on how to do this and that reproduces the figures below, can be found here.

The figure below demonstrates the potential loss of information if we ignore the timing information about the irregular sampling of the data. The figure compares the Short-time Fourier Transform (STFT) based spectrogram (MFCC), which assumes regular sampling, with Lomb-Scargle based spectrogram, which takes dt as additional input. In the top row, we see the signal plot over time-steps (left) and over time (right). This exaggerated synthetic example makes it obvious that the dt can have a significant impact on the signal spectra and thus the spectrogram that does not take the dt into account (bottom left plot) can not detect the true spectral representation of the signal. In the bottom right, the Lomb-Scargle spectrogram is able to capture the change in the signal well.

For further details and more advanced applications of Lomb-Scargle method refer to the following links:

What are interpolation networks?

Researchers dealing with time series data in EHRs are developing data modelling methods that can make use of extremely sparse and irregular data. These models are usually evaluated on classification or regression tasks like predicting the onset or probability of sepsis based on the physiological measurements of a patient.

The common approach involves two stages where the first stage models the trajectories of the irregular and sparse physiological measurements, and the second stage is a common black box prediction neural network.

For example, the first stage in [Shukla and Marlin, 2019] uses a semi-parametric interpolation network [github] to model the data by learning three distinct features: a smooth interpolation for slow trends in the signal, a short time-scale interpolation for transients, and an intensity function for local observation frequencies. An earlier work [Futoma et al., 2017] implements the first stage with multi-task Gaussian processes.

These models attempt to recover the missing information from time-series but only in latent space as the models are trained end-to-end with the prediction neural network on top. However, if your time-series have missing data points, approaches outlined in the section below should be more applicable.

For the scenarios where you have different input channels operating at different frequencies (multi-rate multivariate time series) you should look into [Che et al., 2018b] where the authors deal with this problem using the Multi-Rate Hierarchical Deep Markov Model (MR-HDMM).

For working with physiological signals, it is common to use some form of CNN and/or RNN based architecture [Faust et al., 2018, Fawaz et al., 2019] like 1-D ResNet or architectures based on WaveNet sometimes combined with LSTM or other RNNs.

For efficiently training on sparse, irregularly sampled, and multivariate data the new model called Multi-Time Attention Networks by [Shukla and Marlin 2021] offers great results beating the SOTA benchmarks.

What are Neural Ordinary Differential Equation models?

ODE-RNNs [Rubanova et al., 2019] are a novel model that builds upon the original idea of NeuralODEs [Chen et al., 2018] by generalizing the RNNs to have continuous-time hidden dynamics. In short, they are a natural fit to solve irregularly sampled time series data problems and outperform the RNN counterparts in that domain. This is best summarized in this excerpt from the paper abstract:

“Both ODE-RNNs and Latent ODEs can naturally handle arbitrary time gaps between observations, and can explicitly model the probability of observation times using Poisson processes. We show experimentally that these ODE-based models outperform their RNN-based counterparts on irregularly-sampled data.” [Rubanova et al., 2019]

Figure 1 [Rubanova et al., 2019]

A downside of the ODE based models is that the solution to an ordinary differential equation is determined by its initial condition. The follow-up work [Kidger et al., 2020] deals with this issue using the controlled differential equations and set the new state-of-the-art. The newly introduced Neural CDE outperforms ODE-RNNs and GRU-D models on CharacterTrajectories, PhysioNet, and Speech Commands dataset.

How to make an input feature out of timestamps? ⏱️

Usually, timseries data contains a timestamp for each data point. The first thing to do is to convert the timestamps to delta t-s (dts). Next, you will need to perform a cumulative sum on the dt feature to get a monotonically increasing value. Finally, you might need to standardize the values (scale to the range [0, 1]) if your model always works with uniformly long sequences at the input.

Depending on the problem, if your model is sometimes working on a subsequence of a much longer input stream and needs to be aware of the position in the stream, then computing a cumulative sum of dts beforehand might be required and the standardization needs to be omitted. Otherwise, the model would not receive the contextual information (position in the input stream) which could potentially have a significant impact on the performance of the model.

Finally, the newly computed feature can be provided to the models as additional input.

This method can also be seen as a form of positional encoding of the Transformer to deal with irregularly sampled data. The added dt based feature allows the attention layer to compute the similarity (distance) between the data points using the precise timing information in addition to the other features.

How to deal with missing values in time-series? 🔍

To effectively model the data with missing values it is important to understand the cause of the missing data. For example, if the data is missing at random, the rest of the data is still representative and well suited for modelling. However, if the missing values do not occur at random, the rest of the data will be biased. In statistics, different types of missing data are classified as:

Missing completely at random (MCAR)
Missing at random (MAR)
Missing not at random (MNAR)

To minimize the work required to analyze, identify and deal with these different types of missing values, deep learning methods have been developed that can deal with these issues directly. Such models take the information about the missing values along with the available data and are trained end-to-end for a specific task, resulting in more accurate models that are aware of the statistics of the missing values.

For dealing with missing values in multivariate time series the authors of [Che et al., 2018a] propose the GRU-D model which ingests the explicit representation of missing values (masking and time interval) and incorporates them into the architecture. This way the model can exploit not only the long-term temporal dependencies but also the patterns of missing values.

Beware that ODE-RNNs [Rubanova et al., 2019] outperform GRU-D on some benchmarks.

If you have a vast amount of data, it might also be fruitful to look into the methods for the reconstruction of irregularly and regularly missing data [[Chai et al.]](https://www.nature.com/articles/s41598-020-59801-x#Bib1). This is an analogous approach to the image and audio super-resolution networks. Using this approach you can learn the underlying distribution of your data which is usually a crucial factor in getting a high performing model.

Final note 🎵

To choose the right approach for your specific case it is very important to analyze and understand your own data before getting lost in the rabbit hole of experimentation with all the different approaches that potentially might work. This is only one of the point from the best practice guidelines for deep learning which you should religiously follow if you value your time.

Hope this was helpful and you were able to get some useful pointers and links that will help with your deep learning endeavors. Cheers! 🍻

Nips 2018 Plan And Notes

Thu, 22 Nov 2018 00:00:00 +0000

NIPS 2018

(=NeurIPS 2018)

List of all posters here.

Registration Desk

7:00 am - 8 pm Sunday QR code required for check in
7:00 am - 7 pm Monday - Friday QR code required for check in
7:00 am - noon Saturday QR code required for check in

Food: Breakfast and lunch are not provided.

6:30 pm Monday reception in Room 220 AB and 210 ABCDEFGH
6:30 pm Reception on Saturday evening reception in 220 ABCD and 210 ABCDEFGH

Sunday, 2.12.2018 NIPS EXPO

8:00 - 9:00 am Alibaba Group, Room 511E: Video Understanding in Youku
9:00 - 10:00 am Capital One Services, Room 517D: Eno’s Deep Learning-Powered Natural Language Understanding
10:00 - 10:30 am Coffee break, Rooms 220A and 220B
10:30 - 11:30 am Alibaba Group, Room 517D: AI at Alibaba
11:30 - 12:30 am Ouster, Room 517D: Native Camera Imaging on LIDAR and Novel Deep Learning Enablement
12:30 - 2:00 pm Lunch on your own
2:00 - 3:00 pm (Facebook was full, NAVER I missed and got into wrong room) NETEASE, 511B: Make reinforcement learning in touch with industry: An Efficient, Elastic and Extensive Reinforcement learning Framework
3:00 - 4:00 pm Hudson River Trading, Room 517D: Inside the Black Box: Machine Learning and Optimization at HRT
4:00 - 4:30 pm Coffee break, Rooms 220A and 220B

Monday, 3.12.2018 Tutorials

Tutorials Session 1 Mon Dec 3rd 08:30 – 10:30 AM @ Room 220 CD
Adversarial Robustness: Theory and Practice
10:30 - 11:00 AM Coffee Break
Tutorials Session 2 Mon Dec 3rd 11:00 AM – 01:00 PM @ Room 220 CD
Unsupervised Deep Learning by Alex Graves · Marc’Aurelio Ranzato
Tutorials Session 3 Mon Dec 3rd 02:30 – 04:30 PM @ Room 220 CD
Automatic Machine Learning by Frank Hutter · Joaquin Vanschoren
4:30 - 5:00 PM Coffee break
5:00 - 5:30 PM Opening Remarks
5:30 - 6:20 PM Invited Talk: Laura Gomez: Accountability and Algorithmic Bias: Why Diversity and Inclusion Matters
6:30 - 8:30 PM Opening Reception

Tuesday, 4.12.2018 Invited Talks, Papers and Demonstrations

8:30 - 9:20 Invited Talk: Ed Felten: Machine Learning Meets Public Policy: What to Expect and How to Cope
9:20 - 9:40 Test of Time Award
9:40 - 10:05 Coffee break
10:05 - 10:20 Parallel Tracks: On Neuronal Capacity @ Room 220 CD
10:20 - 10:25 Parallel Tracks: Learning Overparameterized NNs via Stochastic GD on Structured Data @ Room 220 CD
10:25 - 10:30 Parallel Tracks: Size-Noise Tradeoffs in Generative Networks @ Room 220 CD
10:30 - 10:45 Parallel Tracks: A Retrieve-and-Edit Framework for Predicting Structured Outputs @ Room 220 E
10:45 - 12:45 Posters A and Demonstrations
12:45 - 2:15 Lunch on your own
1:15 - 2:15 Town Hall Meeting, Room 517 CD
2:15 - 3:05 Invited Talk: Michael Levin: What Bodies Think About: Bioelectric Computation Outside the Nervous System, Primitive Cognition, and Synthetic Morphology
3:05 - 3:30 Coffee Break
3:30 - 3:35 Parallel Tracks: Evolved Policy Gradients @ Room 220 E
3:35 - 3:40 Parallel Tracks: Answerer in Questioner’s Mind @ Room 220 CD
3:40 - 3:45 Parallel Tracks: Bayesian Model-Agnostic Meta-Learning @ Room 220 E
3:45 - 3:50 Parallel Tracks: Learning to Optimize Tensor Programs @ Room 220 CD
3:50 - 4:05 Parallel Tracks: Generalisation of structural knowledge in the hippocampal-entorhinal system @ Room 220 CD
4:05 - 4:25 Parallel Tracks: @ Room 220 CDE
4:25 - 4:40 Parallel Tracks: A probabilistic population code based on neural samples @ Room 220 CD
. . . . . . . Parallel Tracks: How Does Batch Normalization Help Optimization? @ Room 220 E
4:40 - 5:00 Parallel Tracks: @ Room 220 CDE and 517 CD
5:00 – 7:30 Posters B and Demonstrations

Wednesday, 5.12.2018 Invited Talks, Papers and Demonstrations

8:30 - 9:20 Invited Talk: Joelle Pineau
9:20 - 9:45 Coffee break
9:45 - 10:05 Parallel Tracks: @ Room 220 CDE and 517 CD
10:05 - 10:20 Parallel Tracks: Exploration in Structured Reinforcement Learning @ Room 220 CD K 10:20 - 10:45 Parallel Tracks: @ Room 220 CDE and 517 CD
10:45 - 12:45 Posters A and Demonstrations
12:45 - 2:15 Lunch on your own
2:15 - 3:05 Invited Talk: Ayanna Howard: Investigations into the Human-AI Trust Phenomenon
3:05 - 3:30 Coffee Break
3:30 - 5:00 Parallel Tracks: @ Room 220 CDE and 517 CD
3:50 - 4:05 Parallel Tracks: Policy Optimization via Importance Sampling @ Room 220 CD
4:05 - 5:00 Parallel Tracks: @ Room 220 CDE and 517 CD
5:00 – 7:30 Posters B (our slot!) and Demonstrations

Thursday, 6.12.2018 Invited Talks, Papers and Demonstrations

8:30 - 9:20 Invited Talk: David Spiegelhalter: Making Algorithms Trustworthy
9:20 - 9:45 Coffee break
9:45 - 10:45 Parallel Tracks: @ Room 220 CDE and 517 CD
10:45 - 12:45 Posters A
12:45 - 2:15 Lunch on your own
2:15 - 3:05 Invited Talk: Kunle Olukotun: Designing Computer Systems for Software 2.0
3:05 - 3:30 Coffee Break
3:30 - 5:00 Parallel Tracks: @ Room 220 CDE and 517 CD
5:00 – 7:30 Posters B

Friday, 7.12.2018 Workshops

All workshops are scheduled for: 08:00 AM – 06:30 PM
10:30 – 11:00 Coffee break
12:00 to 2:00 Lunch on your own
3:00 – 3:30 Coffee break

Saturday, 8.12.2018 Workshops (full schedule)

All workshops are scheduled for: 08:00 AM – 06:30 PM 10:30 – 11:00 Coffee break 12:00 to 2:00 Lunch on your own 3:00 – 3:30 Coffee break 6:30 - 10:30 Saturday night reception and dancing.

How to pip install from private github repository in Dockerfile

Wed, 04 Oct 2017 08:58:00 +0000

This is a short note on how to effectively pip install requirements that contain private repositories in a Dockerfile.

For the security reasons it is a good idea to generate an ssh key specificly for the repository you want to pull with only read permissions. If you refuse to generate a new ssh key, you can transfer existing key through local webserver and remove it after installing the requirements. Details of this method can be read here. Generating new key (more details):

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

After generation add newly generated public key to github repository by going to repository settings, Deploy Keys sidebar, Add deploy key button (more details).

Since the key has only read rights, it is safe to push it to your repository. Next adapt your Dockerfile to copy the keys from the repository:

WORKDIR /root
RUN mkdir /.ssh
ADD local_repo/keys/repo-github-deploy-key /root/.ssh/repo-github-deploy-key
ADD local_repo/keys/repo-github-deploy-key.pub /root/.ssh/repo-github-deploy-key.pub

Finally start the ssh agent, add keys, and pip install like usual:

WORKDIR /code
ADD local_repo/requirements_frozen.txt requirements.txt
RUN \
  apt-get -qy upgrade git openssh-client && \
  chmod 600 /root/.ssh/repo-github-deploy-key && \
  eval $(ssh-agent) && \
  echo "Host github.com\n\tStrictHostKeyChecking no\n" >> /root/.ssh/config && \
  ssh-add /root/.ssh/repo-github-deploy-key && \
  ssh-keyscan github.com >> ~/.ssh/known_hosts && \
  pip3 install -r requirements.txt --src /usr/local/src

It is important to start the agent and add keys in the same RUN command, otherwise it will fail.

If you never added github repository to you requirements here is an example requirements.txt:

Django==1.10.6
django-allauth==0.24.1
# ...
-e git+ssh://git@github.com/username/private-repo-name.git@c0m1th4sh#egg=app-name

Quick Guide To Restoring Your Bootloader

Sat, 12 Jul 2014 21:55:00 +0000

So you installed a new operating system and suddenly your boot loader is gone. In most cases the new OS is aggressive enough to make your computer boot directly to it without even making an effort to give you options for you to boot to your old OS. Worry no more, we are going to fix that now.

I will assume that have installed some sort of Linux and Windows version together, probably some newer Windows (very aggressive with BIOS boot time optimizations). First question: To which of your installed operating systems can you boot now?

Click on your answer to go to the next part:

Windows
Linux
None!

Windows

If you can boot to Windows next part is enabling the Windows bootloader and extending it so it gives you the option to boot into other operating systems. To do that we need to download a small bootloader modification tool called EasyBCD. You can download it here: https://neosmart.net/EasyBCD/

To add Linux to your Windows bootloader menu, do the following steps:

Download and install the EasyBCD software. After completing the installation process run the program.
To add new operating system to your boot loader menu, you want to click on the third button from the top Add New Entry. A new options panel should appear on the right side of the application window:

If you want to add a different Windows operating system to your boot menu, just choose the correct version and the location of the operating system in the Drive drop down menu. The same applies for adding the Mac operating system to your windows boot menu.

For adding Linux operating system, click on the “Linux/BSD” tab and proceed to the next step.
In the Linux tab we first need to select the correct bootloader Type in the drop down menu.

If you do not know exactly the type of your bootloader, chances are that you installed some newer version of Linux that has GRUB 2 as its default bootloader. So open up the drop down menu and select GRUB 2.

NOTE: Do not forget to change the name of your Linux distribution. You probably want to name it “Ubuntu” or “Linux Mint” or whatever you have installed instead of the default name.
For the GRUB 2 bootloader, the Drive location does not need to be specified as we are presented with the option of “Automatically locate and load” the GRUB 2 bootloader. This is very convenient as we don’t need to know or remember the location of the bootloader or which of the operating systems we installed first.

NOTE: If the situation is more complex and you have different Linux flavors installed with each having different versions of bootloader then this option may not work so smoothly

Click on the Add Entry button and a success message on the bottom of the window should appear:
At this point you can close the EasyBCD software and restart your computer. Upon booting you should be presented with a boot menu looking like this:

As you see I have renamed my linux boot option appropriately to Linux Mint.

Do you like this Windows bootloader? If yes, you are finished! You now have a functional bootloader and you can successfully boot into any of your operating systems.

If you don’t like it and want your Grub (or any other) bootloader back, just boot into your Linux system and proceed reading the next part.

Linux

So you can boot into Linux, great! Now let us restore the bootloader and/or insert the options for other operating systems to boot in.

If you booted to Linux by going through Windows bootloader and you prefer the GRUB as your primary bootloader, you need to restore GRUB by reinstalling it on primary partition.

Reinstalling GRUB 2

Run the following command from the terminal:

sudo grub-install /dev/sd?

? is the drive letter on which you want GRUB to write the boot information. Normally users should not include a partition number, which would produce an error message as the command would attempt to write the information to a partition.

If you are using only one hard drive or the operating systems are located on the first primary drive, you will probably use the command on drive “sda” like this:

Example: sudo grub-install /dev/sda

This will rewrite the MBR information to point to the current installation and rewrite some GRUB 2 files (which are already working). Since it isn’t done during execution of the previous command, running sudo update-grub after the install will ensure GRUB 2’s menu is up-to-date.

After doing these steps your GRUB bootloader should appear first after rebooting the computer.

Adding other operating systems

If you are making GRUB your primary boot loader, you probably want to add all other available operating systems to the boot menu.

This part is easy as it does not require downloading or using any additional tools. Only thing that we need is a text editor and your grub configuration file path which is by default: /boot/grub/grub.cfg

Open up the grub.cfg file in any text editor (like gedit) and find the “menuentry” part. Because I have Linux Mint installed, on my machine it looks like this:

menuentry 'Linux Mint 15 Cinnamon 64-bit, 3.8.0-19-generic (/dev/sda1)' --class linuxmint --class gnu-linux --class gnu --class os {
	recordfail
	gfxmode $linux_gfx_mode
	insmod gzio
	insmod part_msdos
	insmod ext2
	set root='hd0,msdos1'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  77a4962b-fb54-4054-ae57-c2292a4fae45
	else
	  search --no-floppy --fs-uuid --set=root 77a4962b-fb54-4054-ae57-c2292a4fae45
	fi
	linux	/boot/vmlinuz-3.8.0-19-generic root=UUID=77a4962b-fb54-4054-ae57-c2292a4fae45 ro   quiet splash $vt_handoff
	initrd	/boot/initrd.img-3.8.0-19-generic
}

To manually add a Windows boot option you want to add the following text to your file just after the previous part:

menuentry "Windows 8" {
	set root=(hd0,3)
	chainloader +1
}

As you see I have added a menu entry named “Windows 8” and the very important part here is set root=(hd0,3) which means to boot the windows operating system from the hard drive 0 (which is the first and only one I have on my notebook) and that it is located at the third(3) partition. For more understanding here is how my disk looks like:

For example if you need to boot your windows XP which is located on the second hard drive, on the first partition you would want to add to grub.cfg something like this:

menuentry "Windows XP" {
	set root=(hd1,1)
	chainloader +1
}

After adding the new entry in the grub.cfg file, don’t forget to save it! Next time I booted my computer I was presented with boot menu looking like this:

If you want to use windows bootloader as your primary boot menu, you probably don’t want the GRUB bootloader appearing right after you choose Linux in your Windows bootloader. To disable the GRUB boot menu, do the following steps:

Open the file /etc/default/grub with text editor with elevated rights. You can do that by opening the terminal (Ctrl+Alt+T) and executing:
```
sudo gedit /etc/default/grub
```
Find the line that contains the GRUB_TIMEOUT variable and set its value to zero. It should look like:
```
GRUB_TIMEOUT=0
```
Do not forget to save the changes! Next you need to open the terminal once again and execute the following command:
```
sudo update-grub
```
This should apply the changes and completely disable the display of the bootloader menu.

By deciding to have GRUB as your primary boot loader, you probably want to skip the windows bootloader after selecting the windows in the GRUB boot menu. In the process of restoring your GRUB bootloader you may have needed to enable the windows bootloader but now when you got what you wanted, the appearing of the unnecessary windows bootloader is just plain annoying. Disabling it is quite easy. Just follow the steps below:

Boot into your Windows and run the EasyBCD software once again. This time click on the “Edit Boot Menu” button.
The following menu should appear on the right:

To completely skip the windows bootloader menu select the “Skip the boot menu” in the “Timeout Options” list under the list of menu entries. Additionally you may want to delete the unnecessary entries in the list by selecting them and clicking on “Delete” button.

Finally just click the “Save Setting” button and you are all done!

After restarting there should be no more windows bootloader menu.

I can’t boot to anything

In this case it is advisable to make yourself a bootable Linux USB drive (I recommend https://grml.org/ but Ubuntu will work fine) and boot into a live Linux on your computer. Then you could try to fix your GRUB bootloader as described in the Linux section above.

~dsalaj/com/

Instead of more photos, make an audio album of your loved ones!

Story time

A necessary warning

Similarity search and Deduplication at scale

Challenges and constraints

Heterogeneity of data sources

Controlling the computational cost

Identifier based matching

Attribute-based matching

Deep learning approaches

Pruning and Sparsification of Neural Networks

Types of sparsity

When to prune?

State of the art

Summary

All about Positional Encoding

What is positional encoding?

What is Absolute Positional Encoding?

What is Relative Positional Encoding?

What is RNN based positional encoding?

Positional encoding for tree-structured data

How to do positional encoding for irregularly sampled data?

How to migrate binary bloat out of git repository

Dealing with symptoms directly

Migrating to git lfs

Pruning and migrating out of git

Corrupt, sparse, irregular and ugly: Deep learning on time series

What is irregularly sampled data or sequence of non-equidistant steps? 📊

Why not just interpolate? 💡

Solving the problem by ignoring the problem 🙈

What are the methods for working with irregular or sparse time series? ⚗️

How to use Lomb-Scargle Periodograms?

What are interpolation networks?

What are Neural Ordinary Differential Equation models?

How to make an input feature out of timestamps? ⏱️

How to deal with missing values in time-series? 🔍

Further reading 📚

Final note 🎵

Nips 2018 Plan And Notes

NIPS 2018

Sunday, 2.12.2018 NIPS EXPO

Monday, 3.12.2018 Tutorials

Tuesday, 4.12.2018 Invited Talks, Papers and Demonstrations

Wednesday, 5.12.2018 Invited Talks, Papers and Demonstrations

Thursday, 6.12.2018 Invited Talks, Papers and Demonstrations

Friday, 7.12.2018 Workshops

Saturday, 8.12.2018 Workshops (full schedule)

How to pip install from private github repository in Dockerfile

Quick Guide To Restoring Your Bootloader

Windows

Linux

Reinstalling GRUB 2

Adding other operating systems

Disabling the GRUB boot menu

Disabling the Windows boot menu

I can’t boot to anything

Migrating to `git lfs`