Why ROC? — Intuitive Explanation

Leave a reply

I’ve often found myself understanding concepts much better when I understand the geometrical interpretation behind that concept. This is especially true for a lot of Math, Machine Learning and AI (although there are definitely a lot of abstract concepts in this field too that often don’t have a geometrical interpretation associated with them).

There might be many posts out there on RoC and what it is and how it’s used to determine the probability threshold for a classification problem, but how is the geometry of the problem changes when we pick another threshold value is something I’ve never encountered anywhere. And this post, I am going to try and intuitively explain everything. ELIF, or what’s the point, right?

Problem Statement

Let’s start with a simple example that’s often used. Let’s say we are faced with predicting whether a tumor is benign or malignant, and assume we have a bunch of features available to us to do this task. For illustration purposes, I am going to keep the graphs and equations really simple, because the point it to understand the concept to be able to apply it later in more complex scenarios.

So, we go ahead and build a classification model and the line/curve it comes up with to separate the categories is shown below -

So, as you can see above the line does a pretty job of classification. The accuracy is high too. So, do we care about anything more? Yep! In the case above, there are two malignant tumors that have been classified as benign. And that’s not a good thing for this particular problem. It depends on the problem really, whether we want our criteria of success to be overall accuracy or true positive rate or false negative rate, etc. For a problem like above, we want to definitely classify as much malignant tumors correctly as possible without caring much about overall accuracy or whether we misclassify some more benign one as being malignant. This is because the goal of such a classification problem should be to identify all malignant cases and intervene for patient care.

So, for problems where we consider both classes to be of equal importance we can consider 0.5 probability as our threshold value (anything above or equal to 0.5 is one class and anything below is another class). But for problems like above, if we can choose a threshold such that it also covers the left out malignant ones, that would be great. And as I will show, choosing a different threshold basically means to move our curve/line of separation a little bit.

But before that here’s some math. You will note the following things from the image below -

Curve/line of separation corresponds to z = 0, which happens at probability of 0.5. For ease of illustration purposes y = x has been chosen
Region above curve/line of separation is where z is +ve and corresponds to region where probability is larger than 0.5. Illustrated by point (3, 5)
Region below curve/line of separation is where z is ‑ve and corresponds to region where probability is less than 0.5. Illustrated by point (5, 3)

So, as seen from above probability of 0.5 corresponds to like y — x = 0

What about probability of 0.4 or less? What does that mean geometrically?
So, we know that anything less than 0.5 means we’re talking about the region below the line. A probability p < 0.5 means that z is negative. For illustration purposes, let’s just assume some probability p(<0.5, maybe it’s 0.4 or lesser) such that value of z is ‑0.7 (we can see from sigmoid function that z will be negative when probability goes below 0.5). What does z = ‑0.7 mean for the curve?
Well, yes, of course it means y — x = ‑0.7 or y = x — 0.7, which is basically another line shifted down a bit from the original line, as shown below -

So, essentially setting threshold to anything but 0.5 shifts the line or curve. And now, we can see the benefit of such shifting — the previously malignant tumors which were classified as benign are now being classified correctly. So, the idea is to shift the line/curve such that it captures what we want to do for our problem — in this case we wanted to increase True Positive Rate, without being too nosy about overall accuracy. So, now the question is — how do we actually pick a good threshold? And that is exactly what the RoC (Receiver Operative Curve) lets us do.

RoC Curve

Now that we know why we have reached here, the idea is pretty simple — try out a range of probabilities (shift your curve/line a lot above z = 0 and below z = 0) and for each probability, capture the True Positive Rate (generally you want this to increase) as well as the False Positive Rate (generally you want this to decrease). Now, plot these out. Something like below, and choose the value of probability threshold that makes more sense for your problem. Also, you can use RoC curve to find out AuC (Area Under the Curve) to get a sense of how your model is doing. You can also plot out various models’ RoC curves and then see which model and which probability threshold makes most sense for your problem -

Cross Validation — Intuitively Explained

Leave a reply

Cross Validation is not a difficult topic. But when it comes to understanding how to get the tuning parameter using Cross Validation, a lot of people get confused. Hopefully, this blog might help out a little bit.

Let’s start from the beginning.

What is Cross Validation?

Cross Validation is a general technique used to identify the better performing model out of a bunch of given models.

Let’s say we have some data and we divide it into train and test,
something like this —

But, why only this 25% is used to test? Why not the starting 25%? Even if we randomly take a certain 25% data, then why only that? The point is that if we train a model with a certain 75% data and use a certain 25% data for testing, then we’ve introduced data bias in our model — it works well for ‘that’ 75% and ‘that’ 25%. It also begs the following questions — is only a certain 25% of data good for testing and only a certain 75% data good for training?

Wouldn’t it be better if we somehow get to leverage the whole data set for testing as well as training?

And this is where K‑fold Cross Validation comes into play.

Basic Idea

Let’s say we have some data like so -

D = {x₁y₁, x₂y₂, .… x_ny_n}

and some ML models m (1, 2,..c)
By the way, are these models yet? I guess not! These are not ready yet to do any predictions. They are simply a configuration of predictor and feature variables. They will only become a models once they pass through a data set. So, these m algorithms can be any mix of algorithms that you think should solve your problem — like for classification problem these can be logistic regression, SVM, Neural Nets, etc.

Anyhow, so here’s what K‑fold CV does -

Step 1 — Set some value of K. 5 or 10 are very common choices. Now, shuffle/permute the data randomly once.

Step 2 — Split the data into K folds (let’s say 5).

Step 3 — This is just for illustration purposes. Since there are k‑folds (5 in this case), each ML algorithm will go through k iterations of training — testing

For each of the iterations shown above, we have a test set and the rest is training set. Now, for each algorithm m in your set —

Train the model on Iteration 1 training set and get the error by using its test set.
Train the model on Iteration 2 training set and get the error by using its test set.

.…. and so on for K iterations. Now, find the average error as (1/k) x (sum of all errors). Note that these will be misclassified cases for classification problems and residuals for regression problems. Instead, we can also find out accuracy for classification problems.

We repeat the above for all our algorithms and choose the one that has lowest average error (or highest accuracy).

So, what do we gain by this exercise? Just this — If there was going to be some bias in our final model due to data selection bias, then we have gotten rid of that and have hopefully selected a better model.

How can K‑Fold CV be used for Complexity Parameter?

When we build a tree, we run the chance of building a big, overfitted tree. Think about it, if there’s nothing stopping tree generation then it will try to fit the train data as best as possible. Overfitting would simply mean that prediction might not be good for test data. So, how do we go around this problem? Well, we need to ‘prune’ our trees.

And so what’s pruning? It’s merging of nodes to make the tree shorter. As shown below, the tree is pruned at t2 node.

As you can guess, the more pruned a tree is, the more Sum of Squared Error (in case of regression trees) or more misclassification error (in case of classification trees) it would have — tree with just node would have the most error (most underfit tree) and the largest tree would have the least error (most overfit tree). The job of pruning is to find the balance here — so that we are able to identify a tree model that does not over or under fit. And pruning does this by something called Cost Complexity Pruning.

In simple words, the idea is this — we add this extra term α|T | (Tree Cost Complexity) to the total cost and we seek to minimize the overall cost. |T| is the total number of terminal nodes in the tree and α is the complexity parameter. In other words, this term is basically penalizing big trees — when the number of leaves in the tree increase by one, the cost increases by α. Depending on the value of α(≥ 0) a complex tree that makes no errors may now have a higher total cost than a small tree that makes a number of errors! This is how this term enables us to find a good tree.

Also, convince yourself that for a given α, there can only be one tree T that will minimize the loss function.

But. How to find α?

And this is where CV comes in. Here’s what the APIs do as an overview (of course, there would be differences in implementations and more subtleties but the point here is to get the intuition right) —

We take a range of α’s we want to try out. For each of there α’s and a given data set and pre-determined value of K(for CV), we do the following —

For α1, find the best tree for Fold 1
For α1, find the best tree for Fold 2
..and so on, For α1, find the best tree for Fold K
For α2, find the best tree for Fold 1
For α2, find the best tree for Fold 2
..and so on, For α2, find the best tree for Fold K
…and so on, till whatever number of α’s you want to try out (typically trying out anything between 30 — 50 are probably enough)

So, for each α, we find the average accuracy of the model (accuracy in case of classification models or RMSE/SSE in case of regressions models. Let’s go with Accuracy for illustration purposes). We can plot this out like below (in practice you won’t need to plot it out and the CV api can tell you the best value of alpha) and see for what value of α we get the highest accuracy and we choose that α and plug in our original tree’s cost function and build a good tree! Note that CV is not generally used for Random Forests or GBM kind of algorithms — this is because CV won’t be much effective there since these algorithms already have a lot of randomness built into them, so chance overfitting of greatly reduced.

Also, note that the shape of the graph below is such because when value of α is lower then it will favor bigger trees and accuracy will be high (if not highest), and when it’s value keeps getting bigger then it favor more and more short trees, the shorter the tree the lesser it’s accuracy. A sweet spot is, therefore, somewhere in the middle.

So, once we find our α, we can use it to build our new tree model rather than using arbitrary parameters like minbucket, etc. to limit our trees.

Think about what CV did here — we had a bunch of models we built using different values of α, we used CV on them to find a good value of α and then eventually use α to find a good single tree model. So, this is just an application of what CV does in general — to find a good model, given a bunch of models.

Understanding GBM intuitively

Leave a reply

A few years back, I was assigned the task of Predicting the probability of students dropping out of college so that there could be some intervention beforehand to help students at risk.
For this exercise I had to first build a Person Matching Algorithm between credit card data and the students data to be able to find out the financial condition of the student’s family — since a lot of times this is also a factor for dropping out. But that’s a topic for another day.

In this post, I am going to write a little bit about the main ML algorithm I used — GBM or Gradient Boosted Machine learning. It’s also one of the top algorithms used in Kaggle competitions and is pretty reliable. When I used this algorithm, I was already well-versed with Decision Trees, Random Forests, etc. but I still had a hard time getting the intuition behind GBM and why it would work and why it was designed the way it is. In this post, I am going to try to break this down for the reader, but the post assumes that the reader is already familiar with how Decision Trees (Regression as well as Classification) are grown.

Before we dive into GBM, we can discuss a little bit about how the world got here -

Single Trees — First came the Decision Trees. These are single trees made from training data and are only good for visual representation of the problem. Never use them in production because they are very sensitive to data and the prediction values change very easily for even slight variations in training data. Also, definitely not use simple trees when data is skewed in favor of one class — it will almost always predict that same class.
Bagging — Due to problems arising from single trees, people started to look at — well, what do people look at when single things don’t work well? Create many models and average out the result. That’s called Bagging. It comes from the phrase ‘bootstrap aggregating’. There are two techniques involved here — bootstrapping (of data) and aggregating (or averaging of models). In the bootstrapping phase, a number of training data sets are created by ‘sampling and replacement’! What does it mean? Well, it simply means that a data set can be created from original data set for training where a percentage of records will be randomly sampled and since sampling is not excluding anything (even records drawn earlier), there could be a lot of records that are repeated. And we create many such data sets — as you can see this introduction of randomness in data basically reduces the possibility of overfitting. That was for the datasets. Now, for the model — instead of choosing just one model, one can create a number of models with certain variations. Now, these m number of models can be fitted with m number of new datasets and result averaged. Random forest is basically just this. To create the different models, when it builds each tree, for each split only a certain number of features are chosen randomly.
Boosting — Random forests work well. The randomness in data and randomness in models mean that the chance of overfitting is greatly reduced. But does that necessarily mean a good model? Not really! And the reason is quite simple. Random forest is basically taking a majority vote (or average value in case of a ‘regression trees’ forest) from a number of independently created models. That’s right. The trees in RF are not related to each other at all. So, there’s no chance that they are learning from each other’s mistakes. That’s where Boosting comes in! It’s a technique where a number of models are created and all are used to get the final vote (or value), but the each new model is built by improving on the previous model. So, boosting is basically saying — ‘I have this stupid initial model, which is basically an average model. Now, let’s build another model/tree based on this such that it’s a little better. And let’s keep doing this till we (let’s say) build 100 models’!

In short, here’s what GBM does — it computes an initial single prediction value. Then it computes residuals for predicted and actual values, then it builds a tree to predict the residuals. Now, it replaces the initial predictions with new ones — with new prediction value being: initial prediction value + predicted residual value. Once we get new prediction values, we again calculate new residuals and build another tree to predict these residuals. We keep doing this for M number of trees, where M can be user input. As you can see, any tree that’s generated depends on earlier prediction value. This is why GBM is said to be additive and it’s an ensembling technique. This is also the main idea behind boosting — using previous predictions to come up with new models. In our case, if you see closely, we build our first tree based on the residuals from the first prediction, we build our second tree based on the residuals from first tree, and so on.

Here’s what GBM looks like -

This can seem daunting at first, but it’s not.

Here’s step 0 — Let’s assume we have some training data available to us of the form (x, y), for i=1 to n, where the y is some regressor (let’s just take regression as an example to understand GBM. Classification is similar but needs some transformations between steps). x here is not one feature but all values for all features for a single record and n is the total number of records. Here’s a visual representation of this data -

Now, let’s get to step 1.
This step is asking us to calculate f0(x). This is a single value. As a matter of fact, GBM starts with a single prediction value. If you were given a bunch of data and asked to give a single value that you would predict for that data, what would it be? You’ll simply take average of y values from training data and for any incoming new data, this is what you’ll predict. As we’ll see, f0(x) here, for regression is basically the average of all y values in the data.

L is the loss function that we need to minimize. Nothing new here, all ML algorithms minimize a loss/cost function. Input for L is y values from training data and the predicted value γ. Note that since we only need one prediction value, this is going to be same for all for all records. Now, for regression loss function usually taken is — 1/2 x (actual — predicted)^2

There are other Loss functions too for regression that one could use but let’s go with sum of squares of residuals, which is also very common. The Loss function must be differentiable and must be convex so that we can apply gradient descent to it. And we know that sum of squares of residuals follows these properties. Below is a rough graph of one feature variable based loss function against prediction values. The shape of the curve is such because of the square term that we have in our loss function. Note that now we need to take derivative of it and set that to 0, to get to the point where we have the pink line touching the graph horizontally, since that’s the point where the value of loss function would be lowest.

If you compute the derivate, it will come out to be -> -(actual — predicted). Now, let’s say we have 3 records where y was 1, 4, 5. Putting these in the equation and setting it to zero we get, we get -> -(1 — predicted) + (-(4 — predicted)) + (-(5 — predicted)) = 0. Solving this for predicted, we get ->

predicted = (1 + 4 + 5) / 3
So, f0(x) is nothing but average of all y values.

Let’s move to step 2.
In this step, M is the total number of trees we want to build. This is usually a user input, like 100 (or 500!).

All 2(a) is doing is calculating the residuals. The derivative you see is the same as in step 1. Note that to calculate the residuals its simply plugging in values from previous prediction (f = f of m‑1) and derivative of L comes out to be -(actual — predicted). Note the minus sign for the overall value? That’s to get rid of the minus sign in -(actual — predicted). And what’s ‘predicted’ value? It is simply f0(x) when m = 1. It will be f1(x) when m = 2 and so on.

In 2(b), we are building a regression tree to predict the residuals from 2(a). The tree is built with a set number of nodes (or set number of terminal leaves) so that we don’t end up growing super large trees. Note that j in this step is leaf index and m is tree index.

In 2©: So, in 2(b) we made a regression tree. But, it might happen that in a number of cases the leaves we get have more than one values in them. So, then what’s the one value for a given terminal region R? Its simple — it’s the average of the values present in that terminal leaf/region! 2© simply updates the tree with final terminal region values. This is why the equation in this step resembles the equation from step 1, because all it’s doing is averaging!

2(d): This does the new prediction. All it’s doing is — taking previous model’s prediction value and adding to it the residual value to get new prediction values for each records.

Note that a learning rate is also usually applied to residual value (something between 0 and 1), so as to lessen the effect of variance.

Once 2(d) is done, we go back to 2(a) and calculate residuals and then predict those residuals and so on, till M.

Once step 2 finishes, our model (step 3) is ready and prediction.

By no means it’s an easy algorithm to grasp, but hopefully you can start with a a very simple data set and work through these steps as explained to get more understanding of the algorithm.

Simple Single Document Summarizer

Leave a reply

A few years back I created a Single Document Summarizer — a statistical heuristics driven Java application (with GUI) that takes in certain text and summarizes it. In this blog post, I am going to discuss some techniques I used for summarization. These are rudimentary techniques but still work well.

Code can be found at — https://github.com/vaibkv/Single-Document-Summarizer (might have a few repetitive directories as well)

Sentence extraction is done via regex, after which normal NLP preprocessing is done — Stemming (using Porter’s rule-based algorithm) and Stop words removal. After tokenization, we have a sentences to tokens mapping. This is all pretty straightforward. I also used DUC 2002 (Document Understanding Conference) corpus to test the algorithm for it’s efficacy but all that’s for the paper.

Now, let’s discuss the features used to give weights to sentences -

Topic Segmentation Words — If we are able to find important words that correspond to sub topics in the text, then sentences containing those words are probably important and should be given more weight. This is also important to get coverage of the topics written about in the text.

For finding such words we have used tf-isf (term frequence — inverse sentence frequency) along with word density score dens(w).

The tf.isf score is -

The above equation calculates tf.isf for word w in sentence s. stf (w, s)/|s| is the normalized frequency of word w in sentence s where |s| is the total number of words in s. The log term is the inverse sentence frequency. Here, Ns is the total number of sentences in the document and sf (w) is the number of sentences in which w occurs at least once. This gives us a hold over the distribution of the word throughout the document. If a word occurs very often throughout the document then its isf score would be low and if it occurs in only a few places then its isf score would be high. The intuition here is that the words that appear in only a few places are better candidates for topic segmentation.
But there’s a problem. Even if a word appears at very few places (i.e., it has good isf) but the places where it appears are very far off in the text then that word is also not well suited for topic segmentation. The intuition here is that if a word is representative of a sub-topic then it should occur frequently in a specific region and infrequently in other parts. For this purpose we’ll calculate the word density as follows:

Here, occur (k) and occur (k+1) represent the consecutive positions of w in the text and the dist function calculates the distance between them in terms of words. |w| is the total number of occurrences of w in the document. When we sum up these inverse distances we get a higher value for words that are dense in a region and lower values for dispersed words.

Combining these two, we have — tf.isf (w, s) x dens (w)

Sentence Location — The intuition here being that starting and ending sentences are probably important. So, we normalize the sentence positions between 0 and 1 and give more weight to starting and ending sentences.
Position of next sentence — Position of a sentence may also have effect. For example — “Sachin is an excellent batsman. He lives in Mumbai. He has played a lot of cricket. A lot of cricket is played in India”. Here we are talking about a player named “Sachin”. The first sentence is important due to the fact that we continue to talk about “Sachin” in the second sentence. The second sentence also finds its relevance since we continue to talk about “Sachin” in the third sentence too. Now the third sentence is not as important as the first and second ones since the fourth sentence does not talk about “Sachin”. So, let’s give importance to a sentence if the sentence following it refers to it. To have used the approach of identifying cue words like ‘alternatively’, ‘although’, ‘accordingly’ etc. to find such sentences. The formula used is -

weight = number of cue phrases in the sentence / total number of words in sentence

Also, if a sentence has a sentence following it in the same paragraph which starts with a pronoun, then we also add 0.1 to the weight of the sentence.

Title words — If the sentence has words that are used in the title (except stop words) then the sentence maybe indicative about what the text is about. As such, we give more weights to such sentences.
Theme words — We try to find words that are representative of the themes present in the text. For this, we sort the normalized term frequencies and take the first 4 words with highest frequencies. So, now the sentence weight for this feature is -

weight = number of theme words in the sentence / total number of words in sentence

What are normalized term frequencies? For a given term t1, the normalized term frequency tf1 = total frequency of t1i / max frequency

Proper Nouns — It can be said that sentences containing proper nouns have more information to convey. So, we can use this to add weight to sentences too.

weight = number of proper nouns in sentence / total number of words in sentence

Sentence Length — Lengthier sentences contain more information.

weight = number of words in sentence / max sentence length in document

Punctuation — Certain punctuation are important to identify important sentences. For example, exclamation mark (!) may mean some sudden thought or emotion. Similarly, a question followed by an answer should also have good information.

weight = total punctuation in the sentence / total words in sentence

We’ve omitted some punctuation as well as ? and ! have been given more importance (adding 1.0 for each of these)

Numeric Data — Sentences containing numerical data can be important. They can have important statistics.

weight = total numerical data in sentence / total words in sentence

After we have these individual weights for each sentence, we combine them using a linear combination to find total sentence weight: α(sentence location weight) + β(weight due to next sentence) +γ(title word weight) + δ(term frequency weight) + ε(theme word weight) + ζ(weight due to proper nouns) +η(weight due to cue phrases) +θ(weight due to topic segmentation words) +ι(weight due to sentence length) +κ(weight due to punctuation) +λ(weitght due to numeric data), where the greek letters have weights between 0 and 1 and can be tweaked to influence the weight of features.

Once this is done we can select the top x % of ranking sentences and then re-arrange them in summary in the same order they were in the original text. Here, x can be user input.

Minimal Gradle Post | What you need to know

Leave a reply

I’ve been using Gradle for sometime at the workplace and I find it to be a good build tool. In the past I’ve SBT, Maven and Nant as well as, well, MSBuild. It was only MSBuild that I grew much familiar with back in my .net days.

This is a minimal Gradle post for whoever wants to get started with it with little/no fuss. Let’s get right into it.

What’s Gradle used for — It’s a build management tool. What this means is that you can use Gradle to manage building your projects, manage dependencies within projects and external to them, manage how tests are run, write pre/post build tasks/targets, and so on. You get the idea.

Two main ideas in Gradle — Projects and tasks

A gradle build can consist of one or more projects and each project can have one more more tasks. Tasks are nothing but units of work need to be done. A gradle project needs to have at least one build.gradle file. If there is only one such file it needs to be present in the root directory of the project. But you can additionally have build.gradle for each project in your project. Gradle build files contain DSL code and can also contain Groovy or Kotlin code. The DSL is basically a declarative kind of language.

At this point, let’s dive into code and see a few things first hand. I will be using IntelliJ IDEA to set up a Gradle project. You are free to use any editor of your choice as long as it has support for Gradle.

Download IntelliJ, install Gradle plugin (it would be checked by default), Setup a new ‘Gradle’ project. You will see a build.gradle file setup for you as well as a settings.gradle file.

Note that you will also see a few more things — gradlewrapper, gradlew.sh and gradlew.bat files

So, what’s gradlewrapper?
Say you run multiple projects on your laptop. Some might be using SBT, some Maven, some Gradle, etc. Gradlewrapper let’s you use Gradle local to your project by providing Gradle jars that don’t need to be available system wide. Meaning that you won’t need to install Gradle on your system. You can use the Gradle jars and scripts that come with Gradlewrapper and that’s the end of story. To use Gradle through Gradlewrapper you will need to submit your Gradle commands to the scripts that ship with Gradlewrapper — gradlew.bat for windows and gradlew.sh for *nix, much the same way as you would use the ‘gradle’ command from command line.

so, for example a normal gradle command could look something like


gradle

but if you don’t want to install Gradle on your system and rather just use Gradlewrapper, you will write something like


./gradlew

Project listing goes in settings.gradle
settings.gradle will contain at least one project name — the root project, by default it’s the name of the directory containing the main/root project of your setup. You can of course, change this name to anything of your liking. Over time, if you have more projects, their names also go in settings.gradle file.

The Gradle system is rich with plugins. All plugins available can be seen at this url. Plugins provide additional functionality to your Gradle setup. Plugins might have pre-written tasks that you might not want/have to write yourself, apart from lot more functionalities.
For example, if you want Scala to be the language for your project and you’d like Gradle to make src/test directories for the same, you can use the ‘scala’ plugin and ‘refresh’ your build.gradle and you will see the scala related directories in your project structure.
(By the way, you can also do a ‘refresh’ by executing gradle –refresh-dependencies task from command line)

For instance, my build.gradle file for this exercise looks like this —


group 'org.practice'
version '1.0-SNAPSHOT' //when version is set you will have a jar file with 
//name like mylibName-1.0-SNAPSHOT.jar, in this case

apply plugin: 'scala'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    //example on how to target jars in a specific local directory
    //compile files('/home/vaibhav/spark/spark-2.1.0/core/target/spark-core_2.11-2.1.0.jar')
    compile group: 'org.apache.spark',name: 'spark-core_2.11', version: '2.1.0'
    testCompile group: 'junit', name: 'junit', version: '4.11'
}

Couple of things to note above -
1. You can see how to apply a plugin.
2. When you might have setup your sample project(as mentioned above) as a gradle project, you would have been asked to supply groupId and artifactId. GroupId, ArtifactId and Version, also called GAV, is a pretty standard way in the mvn world to identify a library. Gradle follows the same mvn convention and in fact, the gradle project structure by convention, you will find, is much like mvn project structure.

src/main/scala
src/test/scala

3. There would be a sourceCompatibility option meaning that you compile your code to be compatible with a certain version of java run time.
4. Under ‘repositories’, you can list the repos Gradle will hit to search for the dependencies mentioned under ‘dependencies’. You can mention any number of repos by providing a url. MavenCentral is available by default.
You can also specify local file system path to load dependencies from, something like

runtime files(‘lib/project1.jar’,‘lib2/project2.jar’)
There are many ways to specify how to pick, from where to pick, and what to exclude. I am sure you can find the right syntax when you need it.

5. You can see that // works for comments, as well as /* .… */
6. When mentioning what dependencies to download, you provide their GAV. By the way, if you don’t tell gradle where to download your dependencies, it will do so by default at ~/.gradle/caches/
7. You can group your dependencies — above you can see that I don’t need junit in my main project but I need it in my test project. So I use a pre-built group ‘testCompile’. Similarly, for the ‘compile’ group.

If you’d rather not use gradlewrapper, you can install gradle using brew in Mac or apt-get install on Ubuntu, etc. You might need to set GRADLE_HOME if the installation does not already does that for you.

gradle.properties
You can specify your gradle specific settings like what jvm to use, whether to keep gradle daemon alive(recommended setting), etc. in a gradle.properties file. If you are using gradlewrapper, you should already have gradle-wrapper.properties file for you.

Gradle Tasks
So far, we’ve talked about setting up gradle and about projects. Let’s take a look at tasks now.
A build.gradle file can have many tasks, these tasks can be grouped or not grouped. When not grouped, a task is considered a ‘private’ task and will show up under ‘Other tasks’ when seeing the output of command ‘gradle tasks’, which lists all tasks. Here’s an example of a grouped task and an ungrouped task-


task calledOnce {
    println("I will get printed first")
}

task helloworld {
    group 'some group name'
    description 'this is desc for task helloworld'

    doFirst {
        println("hello world task begins")
    }

    doLast {
        println("hello world task ends")
    }
}

> gradle helloworld

> I will get printed first
> hello world task begins
> hello world task ends

doFirst, doLast are phases within a task. Statements in task closure not under any phase by default go into configuration phase and are executed once before any other phases, no matter whether you call that particular task or not.

Tasks can depend on each other
This is a normal procedure for most build scripts. Execution order of tasks depends on which tasks depend on which ones.

example,


defaultTasks 'compile' //since a default task has been specified, on the command line it will suffice to just type 'gradle'

apply plugin: 'scala'

task compile(dependsOn: 'compileScala') {
   doLast { 
       println("compile task")
   }
}

//shorter syntax for defining doLast, doFirst
compileScala.doLast(dependsOn: 'clean') {
    println("compileScala task given by scala plugin")
}

task clean {
    doLast { 
       println("clean task")
   }
}

Note that you can also create your own tasks by extending the DefaultTask class, but this is not something we are going to try in this post.

Multi-project build structure
Usually you would have a few sub-projects or modules in your project and they also need to be built, tests executed for them and there might be dependencies among these sub-projects. Of course, gradle gives you option here. Let’s say you have project structure like this —

root_project
— Web
— Core
— Util

In this case, your settings.gradle would contain ‘include’ something like this —


include 'Core', 'Web', 'Util' //telling gradle we have sub projects

Now, the build.gradle file needs to cater for these sub-projects as well. There will be some tasks that are specific to sub-projects, some tasks common for all, etc. Take a look —



//will be called once for each project, including root
allprojects {
    //groupId: ....
    //version: ....
}

//does not apply to root
subprojects {
    //apply specific plugins, etc.
}

//root_project specific dependencies
project(':org.practice.root_project').dependencies {
    compile project(':org.practice.Core'), project(':Util') 
    compile 'some lib'
}

or,


allprojects {
...
}

subprojects {
...
}

//Core specific stuff
project(':Core') {
    dependencies {
    ...
    }
}

//Util specific stuff
project(':Util') {
...
}

//Web specific stuff
project(':Web') {
...
}

Note that sub-project specific stuff could also go under their own separate build.gradle files.

Publishing artifacts
Published files are called artifacts. Publishing is usually done with the help of some plugin, like maven.


apply plugin: 'maven'

uploadArchives { //maybe some nexus repo or local
    repositories {
        mavenDeployer {
            repository(url:"some url")
        }
    }
}

Hope the above gives you necessary information to get going with gradle! Thanks for reading!

# PIECE OF ZEN #

By Kumar Vaibhav

Why ROC? — Intuitive Explanation

Cross Validation — Intuitively Explained

Understanding GBM intuitively

Simple Single Document Summarizer

Minimal Gradle Post | What you need to know