Vojtech Ruzicka's Programming Blog

Java 12 Raw String Literals

Mon, 26 Nov 2018 22:12:03 GMT

String Literals

Currently (pre-Java 12), string literals in Java are in many cases unnecessarily hard to deal with and have some limitations.

First of all, Java does not support multi-line strings. If you want your string to span multiple lines, you have to concatenate multiple strings:

String myString = "This is my string" +
        " which I want to be " +
        "on multiple lines.";

It gets worse though. If you want your string to actually contain new lines, you need to insert \n after each line.

String myString = "This is my string\n" +
        " which I want to be \n" +
        "on multiple lines.";

Special characters such as newline are escaped with backslash \. This is necessary for double quotes " as they would terminate the string and also for backslash \ character as it would rather indicate escaped character. In addition to that there is a whole bunch of escape sequences in java:

Sequence	Usage
\t	Tab
\b	Backspace (remove previous character)
\n	New line
\r	Carriage return
\f	Form feed
\'	Single quote character
\"	Double quote character
\\	Backslash character

String literals are usually hard to read because they are plagued with escape sequences:

String myPath = "Open \"C:\\Program Files\\Java\\jdk1.8.0_151\"";

It gets especially bad with regular expressions, which can be using a lot of backslashes and are then hard to read because of escaping.

Raw Strings

Turns out, that other JVM languages already solved the problem with multiline strings and readability of escaped sequences. They are called Raw Strings. They can span multiple lines without concatenation and they don't use escaped sequences. You can use backslashes or double quotes directly. For example, in Kotlin, in addition to regular string literals, you can use Raw Strings with three double quotes """ instead of just one.

val text = """
This is my string
which I want to be
on multiple lines.
It can contain backslash \
Or even double-quotes "
"""

On JVM, also Groovy and Scala support raw strings using """. Java is like usually late to the party (similar to introducing JShell REPL) and only now, in version 12, introduces what is already common both in other JVM and non-JVM languages.

Raw String Literals in Java

In Java 12, one of the new features will be JEP 326: Raw String Literals. Unlike Kotlin, Groovy or Python, which use """ Java decided to use backtick ` notation, which is currently used by Javascript or Go.

String rawString = `This is my string
which I want to be
on multiple lines.
It can contain backslash \
Or even quotes "`;

But unlike javascript, backticks inside the string don't need to be escaped. Instead, you can use multiple backticks to start the string and then the same number of backticks to end it:

`This uses single backtick`
``This can contain backtick `, see?``
```Can use any number of backticks```

No matter how many backticks you use or whether you use raw or original string literals, the resulting .class file will be the same and both literals will produce java.lang.String.

Formatting

One problem with multiline raw strings is that they interpret all the characters on each line as part of the string. Imagine the following code.

public class Main {

    public static void main(String[] args) {
        String myString =
        `This is my string
        which I want to be
        on multiple lines.`;
        System.out.println(myString);
    }
}

Each of the lines of the raw string contains some spaces at left as indentation, so the code is nicely formatted - 4 spaces indentation because it is inside a class and another 4 because it is inside the main method. If you run it, you will see this:

This is my string
        which I want to be
        on multiple lines.

The string contains the indentation, which was supposed only to format the code to be more readable, but it is not intended to be part of the string.

A quick fix would be to remove the indentation, but it would be hard to read:

public class Main {

    public static void main(String[] args) {
        String myString =`This is my string
which I want to be
on multiple lines.`;
        System.out.println(myString);
    }
}

There are actually several new methods in the String class, which are useful for handling indentation.

The method align() trims horizontal and vertical indentation of the string and keeps just relative indentation of the lines.

public class Main {

    public static void main(String[] args) {
        String myString = `
        This is my string
        which I want to be
        on multiple lines.
            Note that after alignment,
                only relative indentation is preserved.
        `.align();
        System.out.println(myString);
    }
}

After alignment, the output will be this:

This is my string
which I want to be
on multiple lines.
    Note that after alignment,
        only relative indentation is preserved.

If you don't want to trim whitespace but only increase or decrease alignment, you can use String.indent(numberOfSpaces)

// Increase existing indentation of each line by 2
myString.indent(2);
// Decrease existing indentation of each line by 2
myString.indent(-2);

As a shorthand, if you call align() with an integer parameter instead of with zero parameters, it will align first and then indent by the given number:

// Align and then increase existing indentation of each line by 2
myString.align(2);

String Interpolation

Other languages, which currently support Raw String literals usually also support string interpolation. It is basically a fancy name for substituting variable placeholders in the string with values, such as:

    val name = "John"
    println("Hi, my name is ${name}.")

Unfortunately, this is still not possible in Java, not even Java 12. JEP 326 even explicitly states that this is not intended to be part of this proposal. However, it may be introduced by some other proposal in the future.

IntelliJ IDEA support

The good news is that since version 2018.3, IDEA already supports Raw String Literals.

You can convert from good old string literals to the raw variant and vice versa. IDEA will handle escape sequence conversion and splitting to multiline for you.

IDEA is also able to detect if you are unnecessarily using too many backticks and will be able to remove excess backticks for you.

In addition to this, it now supports actions such as spell-checking or regular expressions detection in Raw String Literals. You can read more in Preview Raw String Literals in IntelliJ IDEA 2018.3.

Try it yourself

As of 11/2018 Java 12 is not out yet, but you can already download early access build of JDK 12 and try Raw Strings for yourself.

Spring Boot Admin Tutorial

Sun, 18 Nov 2018 22:12:03 GMT

Monitor and manage you Spring Boot apps with a nice UI on top of Spring Boot Actuator endpoints.

Spring Boot Actuator

Actuator is a Spring Boot module, which adds REST/JMX endpoints to your application, so you can easily monitor and manage it in production. The endpoints offer health-check, metrics monitoring, access to logs, thread dumps, heap dumps, environmental info and more. I've covered Actuator in depth in the following article:

Actuator: Spring Boot Production Monitoring and Management

03 September, 2018

Monitor and manage your application in production with Spring Boot Actuator 2.x. Gather metrics or check health easily.

Spring Boot Admin

Actuator is powerful and great and it is easy and convenient to consume the endpoints with some other application - you just make a simple REST call. It is not so great when used by a human. For humans, it is much more convenient to have a nice user interface you can use to browse all the monitoring data and manage your application. This is actually what Spring Boot Admin Does. It provides you a nice UI layer on top of actuator endpoints with a few extra features on top.

Spring Boot Admin is not a core module provided by the Spring team, it was created by a company called Codecentric. Still, the code is publicly available on Github and it is free.

Client And Server

Unlike Actuator, Spring Boot Admin actually comes in two parts - Client and Server.

The Server part contains the Admin user interface and runs independently from the monitored applications. The Client part is in the monitored application, which registers with the Admin Server part.

This way, even if our application is down or not working properly, the monitoring server is still up and running. Now imagine you have multiple applications (such as Spring Boot microservices) and each of them can be running in multiple instances. With traditional Actuator monitoring, this is hard as you need to access each of them separately and you need to keep track of how many instances and where are running.

With Spring Boot Admin, each instance of your monitored application (Client) registers with the Server after it starts. Then you have a single point (Admin Server), where you can check them all.

Source Code

The source code with a finished application can be found in this Github repository.

Server Setup

Let's first look into how to setup Spring Boot Admin Server. Let's start with a fresh Spring Boot application. You can easily create one using Spring Initializr. Be sure to include the web module.

After creating the project, the first thing we need is to add the Spring Boot Admin Server dependency:

<dependency>
    <groupId>de.codecentric</groupId>
    <artifactId>spring-boot-admin-starter-server</artifactId>
    <version>2.1.0</version>
</dependency>

Note that even though the project is not created by Pivotal, you can find both the client and server modules for Spring Boot Admin in the Spring Initializr. Next, we need to enable Admin Server by annotating our main application class with @EnableAdminServer:

@SpringBootApplication
@EnableAdminServerpublic class SpringBootAdminServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(SpringBootAdminServerApplication.class, args);
    }
}

And that's it. Now you can run your application and after opening http://localhost:8080/ you should see this:

The server is running, but no clients are registered yet. Let's change that.

Client Setup

Same as with server setup, the first step is to add a proper dependency to an existing application:

<dependency>
    <groupId>de.codecentric</groupId>
    <artifactId>spring-boot-admin-starter-client</artifactId>
    <version>2.1.0</version>
</dependency>

Then you need to define URL where is your Admin Server running. Add this line to your application.properties:

spring.boot.admin.client.url=http://localhost:8080

Adding Actuator

Now you should be able to run both the client and the server. Just make sure there is not a port conflict as both apps by default use 8080. For testing purposes, you can set server.port=0 in your application.properties, so your client will use a random port on startup. This way you can test launching multiple instances running on different ports.

When you open Admin Server UI, you should see your application. When you click the app name, a page with application details should show up.

If you see a screen like above with just a minimum of information, it means that you don't have Actuator as a part of your project. Remember, Spring Boot Admin uses Actuator endpoints under the hood. Fortunately, you need just to add a simple dependency and auto-configuration does take care of the rest.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

However, most of the endpoints are not exposed by Actuator by default. You need to change your configuration in application.properties to expose them:

management.endpoints.web.exposure.include=*

After exposing your Actuator endpoints, you should see much more information in your Admin interface:

For a detailed tutorial on Spring Boot Actuator configuration check this article.

Security

Now when everything is running, we should make sure our Actuator endpoints and the Admin UI are not publicly available to everyone.

Client Security

If you are already using Spring Security, the above will not work for you as the Actuator endpoints are secured by default and your Admin server will not be able to access them. If you don't use Spring Security yet, you need to add a dependency first:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>

For testing purposes, you can temporarily disable Actuator endpoint security by management.security.enabled=false. However, we do want to have security enabled. If you are using basic authentication, you can just provide username and password in your properties file. These credentials will be used by Admin Server to authenticate with the client's Actuator endpoints:

spring.boot.admin.client.instance.metadata.user.name=joe
spring.boot.admin.client.instance.metadata.user.password=my-secret-password

By default, if not configured otherwise, Spring Boot uses default user user and auto-generated password each time your app starts. You can check the password in the console during startup. If you want to provide username and password required by your app explicitly, you can define it in your properties:

spring.security.user.name=joe
spring.security.user.password=my-secret-password

Server Security

Same as with the client, we need to add Spring Security dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>

Now lets configure username and password required to login to the Admin Server in your application.properties:

spring.security.user.name=admin
spring.security.user.password=admin-password

Now in your Client you need to add these credentials as well, otherwise it will not be able to register with the server:

spring.boot.admin.client.username=admin
spring.boot.admin.client.password=admin-password

Now back to the Server part. The last thing we need is to add Spring Security configuration to protect the Admin user interface:

@Configuration
public class SecurityConfig extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        SavedRequestAwareAuthenticationSuccessHandler successHandler 
            = new SavedRequestAwareAuthenticationSuccessHandler();
        successHandler.setTargetUrlParameter("redirectTo");
        successHandler.setDefaultTargetUrl("/");

        http.authorizeRequests()
            .antMatchers("/assets/**").permitAll()
            .antMatchers("/login").permitAll()
            .anyRequest().authenticated().and()
            .formLogin().loginPage("/login")
            .successHandler(successHandler).and()
            .logout().logoutUrl("/logout").and()
            .httpBasic().and()
            .csrf()
            .csrfTokenRepository(CookieCsrfTokenRepository.withHttpOnlyFalse())
            .ignoringAntMatchers(
                "/instances",
                "/actuator/**"
             );
    }
}

What it does, is that it restricts the admin UI only to authenticated users using HTTP basic authentication and form login. The login page itself and static UI resources (javascript, HTML, CSS) are public. Otherwise, you could not log in. Then there is a cookie-based Cross-Site Request Forgery protection. You can see that some paths are ignored in CSRF protection - it's because the Admin Server currently lacks proper support.

Now after a restart, you should see a nice login screen protecting your Admin Server:

Cloud Discovery

Spring Boot Admin client is not the only way to register your applications with the server. Admin Server also supports Spring Cloud Service Discovery. You can read more in the offical documentation or in the Spring Cloud Discovery with Spring Boot Admin article.

Notifications

Once you have monitoring in place, you want to get notified when something goes wrong. The good news is that Spring Admin provides a wide variety of notification options.

If you visit the Admin Server page for the first time, it asks you for permission to display push notifications on your computer. Whenever there is an issue, you'll get a popup message.

Other notifications need a simple configuration. It usually means just providing a few entries in your application.properties. Currently supported services are:

Configuring Email Notifications

If you want to enable email notifications, you need to add Spring email dependency to your Server part:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-mail</artifactId>
</dependency>

Then you need to define your SMTP server, which will be used to send email notifications and credentials. Update your application.properties of your Admin Server.

spring.mail.host=smtp.foo.com
spring.mail.username=smtp-server-user
spring.mail.password=smtp-server-password

Then you need to define recipients and sender.

# Sender email address
spring.boot.admin.notify.mail.from="Spring Boot Admin <noreply@foo.com>"
# Comma-delimited list of recipient email addresses
spring.boot.admin.notify.mail.to=alice@foo.com,bob@bar.com
# Comma-delimited list of carbon copy recipient email addresses
spring.boot.admin.notify.mail.cc=joe@foo.com

Conclusion

Spring Boot Admin offers a nice and useful UI layer on top of Actuator Endpoints. What's more, it allows you to centrally monitor multiple applications with multiple instances, which is invaluable when working in the cloud and with microservices. Make sure though, that you sufficiently protect both your Client and Server. For further information, please check the official documentation.

Reactive Conf 2018: My volunteering experience

Fri, 09 Nov 2018 22:12:03 GMT

Reactive Conf 2018

Reactive Conf is a three-day conference focused on modern frontend and mobile development. I attended the last year for the first time and was impressed, so going this year again was a no-brainer, especially as it moved from Bratislava to Prague, my home city.

There were two stages with parallel talks. In addition to the main stage, there was also Discovery Stage, which was focused not only on presentations, but also on Ask Me Anything sessions with many of the speakers. This was a nice opportunity to ask questions, which in many cases, in my opinion, was even more interesting than the original talks. The questions were managed by sli.do - everyone could submit a question. Then people could vote for their favorite one. Questions with the most votes were then asked.

Each talk was roughly 30 minutes long including some time for questions. The talks were also publicly streamed, so even people not attending the conference could watch. Thirty minutes is a good length in the sense that you get to see more talks and it is easy to stay focused. On the other hand, speakers cannot really go into much detail. It is ok for many talks, but some of them could use more time.

Volunteering

Same as the last year, I didn't come as a regular attendee, but as a volunteer instead. The conference offers a number of "volunteer tickets", which you can apply for. The idea is simple - you get your ticket for free, but you help with running the conference. Of course, not for the whole time. Each day, you dedicate 1/3 of your time for working. The rest of the time, you are free to visit the talks, participate in workshops, attend the parties - anything you want.

The volunteer duties are diverse - registering visitors, helping speakers, dispensing gifts, navigating people around, moderation of questions, and much more. It does not really matter much that you don't get to see every talk. They are recorded and available after the conference.

However, the best part about participating as a volunteer is not about saving money. If it was the case, you could just wait for the youtube videos and watch them from the comfort of your sofa for free. Being there is an excellent opportunity to see the conference from the organizer's point of view. But most of all it is about meeting interesting like-minded people. Sure, you're supposed to network on a conference as an attendee, but I've always found it rather difficult with complete strangers as an introvert. For a volunteer, this is much easier and much more natural. You spend a lot of time with other volunteers from all over the world working together and get to talk to many people during your duties. You get a better chance to approach the speakers and access the VIP zone.

Day 1

The beginning of the conference was the busiest period for us volunteers. We had to register more than 1000 visitors. I spent most of the time providing various goodies to the attendees in exchange for their "Reactive Gift" vouchers.

After the talks finished, there was still a lot more to do. You could stay for a "Discover Czech" event, which was basically a party with a selection of classic Czech delicacies such as nakládaný hermelín (pickled cheese), utopenec (pickled sausages), various Czech pies or traditional plum spirit slivovice. Alternatively, you could join separate parties held by the conference's sponsors in their offices - STRV, MEWS, Kiwi or Socialbakers. For those not interested in partying, there was an opportunity to join a sightseeing tour of Prague.

Day 2

My duties changed on the second day. Now it was mostly about the Discovery Stage - showing the way, letting people in and so on. Then some basic cleanup after the conference.

After the talks, there was a closing ceremony. Then you could choose from a bunch of "Hangouts" with various companies such as Aragon, HackPrague or Green Fox Academy. Alternatively, you could continue in discovering Prague like the day before, which would end with a tasting of local wines, beers and spirits.

Day 3

The last day of the conference was different from the first two days. So-called "Festival day" was not at the same venue. Instead, it was hosted in the Prague Microsoft building. Unlike the first two days, this one was focused mainly on various workshops with limited capacity. It was necessary to register and pay a fee of 20 € in addition to the conference ticket. There were 14 workshops available in total ranging from 2 to 6 hours. Here is the list:

Introduction to Elm
GraphQL with React vs. Angular vs. Vue
Codelab: Build your first Flutter app
Introduction to ClojureScript
Vue.js
Evolving Patterns in React
Building type-safe applications with Flow and GraphQL
Reason DOJO: React and GraphQL
Reason DOJO: Cross-platform Games in Reprocessing
Simplify your data processing with BakeryJS
Introduction to E2E testing with Cypress.io
Service Workers at Your Service
How to build cross-platform desktop apps with React and Electron
Vuex like nobody is looking

In addition to workshops, there was one small stage. In the morning, there were just some recorded talks from the previous year. After that, there were regular talks for the rest of the day.

My duty for the last day of volunteering was moderating questions from attendees to speakers on sli.do for all the talks of the day. This was probably the best of my assignments as I got to see all the talks and it was interesting and kept me occupied.

After the talks and the final panel discussions with some of the speakers, there was an afterparty, still in the Microsoft building.

Impressions

Overall my impression of the conference was very good. Everything was professional and well organized. The event felt world-class, very international, which is not always the case in the Czech Republic. Compared to my last year's experience as a volunteer, everything was even smoother.

The selection of speakers was impressive with interesting and well-known presenters. Except for one questionable talk about "How I met Elon Musk", there was nothing bad.

While 30 minutes (or more like 25) per talk was sufficient in many cases, I was wondering whether it would be better to have some presentations longer. For some topics, it is just not enough time and the talk is more shallow that it could be. I liked that there were lightning talks as a short alternative to the regular presentations. Maybe it would also be good to introduce a longer alternative as well. Then again, some topics were covered in much detail during the workshops on the third day.

While the topics were interesting, I felt that there was too much focus on React neglecting other technologies, frameworks and subjects. At least in my opinion.

I really appreciated that there was a public live-stream and videos published after the conference, so I had a chance to watch the talks I missed because of my volunteering duties.

It was nice that there was a lot to do after the talks - Czech delicacies tasting, after-parties with various companies, discovering the city or Czech traditional live music.

Overall my experience was very positive, I'm sure I'll visit also the next year.

Talks

If you are interested in talks, which were at Reactive Conf, you can watch them all online.

Photos

You can check all the photos from the event or just watch the following 40-second video for a brief overview.

Conclusion

I had a great time volunteering at Reactive Conf 2018. It was really interesting to see a conference from the organizer's perspective and a great opportunity to meet many interesting people from all over the world - the organizers, the volunteers and the speakers. I definitely recommend this as an interesting alternative to just attending a conference.

Faster Development with Spring Boot DevTools

Fri, 02 Nov 2018 22:12:03 GMT

How to speed up your Spring Boot development even more with DevTools and make it more enjoyable and productive?

Setup

As usually with Spring Boot, the setup is really simple. All you need to do is to add the right dependency, and you are good to go. Spring Boot detects this and auto-configures DevTools accordingly.

If you are using Maven:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
    <optional>true</optional>
</dependency>

Alternatively, when using Gradle:

configurations {
    developmentOnly
    runtimeClasspath {
        extendsFrom developmentOnly
    }
}
dependencies {
    developmentOnly("org.springframework.boot:spring-boot-devtools")
}

Note that the dependency is declared as optional. This is important. This prevents DevTools dependency being transitively applied to other modules that depend on your project.

Automatic Restart

Whenever there is a change in files on your classpath, DevTools automatically restart your running application with the new changes applied. When developing locally, this can be valuable as you don't need to redeploy your application manually.

On its own, it wouldn't be so useful as restarts can still take a lot of time. Fortunately, these restarts are way faster than regular restarts because of a clever trick, which DevTools use.

You see, when developing an application, you usually change a class or a few and want to check results in your running application for feedback. You change a tiny fraction of your application as the majority of loaded classes are from frameworks and third party libs.

Under the hood, Spring DevTools use two classloaders - base and restart. Classes which do not change are loaded by the base classloader. Classes you are working with are loaded by restart classloader. Whenever a restart is triggered, restart classloader is discarded and recreated. This way restarting your application is much faster than usual and can be a viable alternative to dynamic class reloading with tools such as JRebel.

Triggering a restart in an IDE

The restart is triggered whenever there is a change on the classpath. However, this varies depending on your IDE. That means, it is not enough to just change your .java files. What matters is that your IDE actually updates .class files on the classpath.

When using IntelliJ IDEA, you need to build your project (Ctrl + F9 or Build → Build Project). You can also configure IDEA to rebuild automatically. Alternatively, you can open your Spring Boot run configuration and define what happens when you trigger an application update (Ctrl + F10):

In the first combo-box, you can select Update trigger file to trigger DevTools restart whenever you call the Update action. Alternatively, you can even select the option to try Hot Swap and restart using DevTools only if Hot Swap failed.

In the second combo-box, you can configure reloading all the static resources and templates when IDEA window loses focus (for example when switching to a browser window).

In Eclipse it is enough just to save your files.

Development only

The usage of the Spring Boot DevTools is intended only for development, not for production. If your application detects you're running in production, DevTools are automatically disabled.

For this purposes, whenever you run your app as a fully packaged artifact such as a jar with an embedded application server, it is considered to be a production app:

java -jar devtools-example-1.0.0.jar

Same applies when your app is started via special classloader, such as on an application server. In contrast, when you run an exploded artifact (such as in your IDE), your application is considered in development mode. The same applies when using spring-boot-plugin to run the application:

Maven:

mvn spring-boot:run

Gradle:

gradle bootRun

Live Reload

LiveReload is a useful tool, which allows you to instantly update your page in browser whenever you make changes in your files such as HTML, CSS, images and more. It even pre-processes files as needed - that means automatically compiling your SASS or LESS files.

Spring DevTools automatically launch a local instance of LiveReload server, which monitors your files. All you need to do is to install a browser extension, and you're good to go. It is not only useful for developing frontend of your application (in case you distribute it as a part of your Spring app artifact), but it can also be used to monitor and reload output of your REST API.

Properties override

When developing your application locally, you usually have different configuration needs than when running in production. One example can be caching. When in production, it is crucial to depend on various caches (such as templating engine's caches, caching headers for static resources and so on). In development, it can make you miserable by serving old data and not reflecting your latest changes. Another example may be enhanced logging, which can be useful in development but too detailed for production.

It is unnecessarily complicated to manage dual sets of configuration by yourself. The good news is that Spring Boot DevTools configure many properties for your local development out of the box.

spring.thymeleaf.cache=false
spring.freemarker.cache=false
spring.groovy.template.cache=false
spring.mustache.cache=false
server.servlet.session.persistent=true
spring.h2.console.enabled=true
spring.resources.cache.period=0
spring.resources.chain.cache=false
spring.template.provider.cache=false
spring.mvc.log-resolved-exception=true
server.servlet.jsp.init-parameters.development=true
spring.reactor.stacktrace-mode.enabled=true

You can check the list of all the properties in the DevToolsPropertyDefaultsPostProcessor.

Remote connection

In addition to local development, you can also connect to a remote application running DevTools. This is not intended for production environments as it can be a serious security risk. However, it can be very useful in pre-production environments.

Enabling remote connection

Remote connection is not enabled by default. You need to explicitly enable it by modifying your pom file:

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <excludeDevtools>false</excludeDevtools>            </configuration>
        </plugin>
    </plugins>
</build>

Or with gradle, you need to set excludeDevtools = false:

bootWar {
    excludeDevtools = false
}

Then you need to set a secret password to be used for authentication when connecting to the remote application

spring.devtools.remote.secret=somesecret

Connecting to a remote app

Once your remote app is running, you can launch a remote connection session. Now all you need to do is to launch org.springframework.boot.devtools.RemoteSpringApplication with URL of your remote app as an argument. Note you should use https if possible.

Running the remote connection is easy in your IDE. In IDEA you need just to create a new run configuration. Go to Run → Edit Configurations... and create a new configuration with the + icon in the upper left corner. Choose Application type.

As the Main class select RemoteSpringApplication from the DevTools module and as a program argument pass URL of your remote app.

After you run this configuration, you should see a similar output if the connection to your remote app is successful.

  .   ____          _                                              __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _          ___               _      \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` |        | _ \___ _ __  ___| |_ ___ \ \ \ \
 \\/  ___)| |_)| | | | | || (_| []::::::[]   / -_) '  \/ _ \  _/ -_) ) ) ) )
  '  |____| .__|_| |_|_| |_\__, |        |_|_\___|_|_|_\___/\__\___|/ / / /
 =========|_|==============|___/===================================/_/_/_/
 :: Spring Boot Remote ::  (v2.0.6.RELEASE)

2018-11-02 17:24:42.126  INFO 16640 --- [           main] o.s.b.devtools.RemoteSpringApplication   : Starting RemoteSpringApplication v2.0.6.RELEASE on DESKTOP-6NJV4ON with PID 16640 (C:\Users\vojte\.m2\repository\org\springframework\boot\spring-boot-devtools\2.0.6.RELEASE\spring-boot-devtools-2.0.6.RELEASE.jar started by vojte in C:\projects\rest-docs-starter)
2018-11-02 17:24:42.130  INFO 16640 --- [           main] o.s.b.devtools.RemoteSpringApplication   : No active profile set, falling back to default profiles: default
2018-11-02 17:24:42.172  INFO 16640 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@3daa422a: startup date [Fri Nov 02 17:24:42 CET 2018]; root of context hierarchy
2018-11-02 17:24:42.679  WARN 16640 --- [           main] o.s.b.d.r.c.RemoteClientConfiguration    : The connection to http://localhost:8080 is insecure. You should use a URL starting with 'https://'.
2018-11-02 17:24:42.800  WARN 16640 --- [           main] o.s.b.d.a.OptionalLiveReloadServer       : Unable to start LiveReload server
2018-11-02 17:24:42.829  INFO 16640 --- [           main] o.s.b.devtools.RemoteSpringApplication   : Started RemoteSpringApplication in 1.212 seconds (JVM running for 1.877)

Once you connect to a remote app, DevTools monitors classpath changes same as it does for local development. However, instead of a local restart, it pushes the changes to the remote server and triggers restart there. This can be a lot faster than building the app and deploying to the remote machine.

Global configuration

You can configure DevTools using configuration properties as you would in any other Spring application. That usually means editing application.properties of your project. This configuration is separate for each application.

However, in some scenarios, it may be handy to have global configuration for ALL the applications running on the same machine. You can create a property file called .spring-boot-devtools.properties located in your $HOME directory. Config declared in this file is applied to all the applications running DevTools.

Limitations

Live Reload

Spring app using DevTools automatically launches a LiveReload server. Unfortunately, only one instance of this server can be running at the same time. To be more precise, just the first one will work. That applies not only to multiple instances of Spring apps with DevTools but to any other apps, which are also using LiverReload under the hood, such as Gatsby in development mode.

If you want to configure your Spring app not to launch a LiveReload server, you can do it in your application.properties:

spring.devtools.livereload.enabled=false

Shutdown Hook

DevTools are dependent on the shutdown hook of SpringApplication. It will not work correctly if you manually disabled the hook using:

springApplication.setRegisterShutdownHook(false);

By default, the hook is enabled, so you don't need to worry about it unless you explicitly disable it.

Collisions with third-party libraries

While DevTools should usually run properly, it may have conflicts with third party libs. In particular, there is a known issue with deserialization using standard ObjectInputStream.

In case of such conflict, you can disable the automatic restart by setting:

spring.devtools.restart.enabled=false

Restart will no longer be triggered. However, the restart classloader will still be used. If you need to disable the classloader completely, you need to do so before launching the app:

public static void main(String[] args) {
    System.setProperty("spring.devtools.restart.enabled", "false");
    SpringApplication.run(MyApp.class, args);
}

Even if you don't use th automatic restart you can still benefit from the other features DevTools provide.

Conclusion

DevTools make your development of Spring Boot applications faster and easier by providing automatic restart and LiveReload functionality. I addition to this, it sets various properties to values more suitable for local development. Furthermore, it allows you to remotely connect to your application and still use most of its features. When running in production, DevTools are not used. For detailed information, see the official docs.

Actuator: Spring Boot Production Monitoring and Management

Mon, 03 Sep 2018 22:12:03 GMT

Basic build information

In one of my previous articles, I described how to obtain some basic information about the current build of your application at runtime in Spring Boot.

Detecting build version and time at runtime in Spring Boot

25 June, 2018

How to obtain artifact version, build time and other build information in a Spring Boot app at runtime?

While this can be handy, it is usually not sufficient. There is a lot more you are usually interested in. And most importantly, it does not involve just the information known at build time, but rather what is the application's current status at runtime. In these situations, you should use Spring Boot Actuator.

Spring Boot Actuator

In short, Spring Boot Actuator is one of the sub-projects od Spring Boot, which adds monitoring and management support for your applications running in production. It exposes various HTTP or JMX endpoints you can interact with.

This post covers Actuator 2.x. If you're using Spring Boot 1.x, be aware that there were many changes in the version 2 and you should look for version 1.x specific configuration instead.

Source code

The source code of example application using Spring Boot Actuator can be found here.

Adding Dependencies

The basic setup is really simple. You just need to add one dependency to your project - spring-boot-starter-actuator.

Maven:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Gradle:

compile("org.springframework.boot:spring-boot-starter-actuator")

As usual, once this dependency is on the classpath, Spring Boot is able to detect this and auto-configures Actuator for you. Just rebuild and start your application.

Up and Running

Now if you run your application, you can check all the available endpoints by accessing /actuator. That means, if you are running locally on port 8080, you'll need to open http://localhost:8080/actuator. You should see something similar to this:

{
  "_links": {
    "self": {
      "href": "http://localhost:8080/actuator",
      "templated": false
    },
    "health": {
      "href": "http://localhost:8080/actuator/health",
      "templated": false
    },
    "info": {
      "href": "http://localhost:8080/actuator/info",
      "templated": false
    }
  }
}

This is the list of all the available Actuator endpoints. As you can see, there is just two of them: health and info. Let's try /health first.

{"status":"UP"}

And now /info. It looks like it is returning just an empty json: {}.

More endpoints

Not very impressive so far. Of course, Actuator offers much more than this. Otherwise, it wouldn't be very useful. There are some examples of available endpoints, with the full list available in the official docs.

Endpoint	Description
health	Application health info
info	Info about the application
env	Properties from environment
metrics	Various metrics about the app
mappings	`@RequestMapping` Controller mappings
shutdown	Triggers application shutdown
httptrace	HTTP request/response log
loggers	Display and configure logger info
logfile	Contents of the log file
threaddump	Perform thread dump
heapdump	Obtain JVM heap dump
caches	Check available caches
integrationgraph	Graph of Spring Integration components

Exposing endpoints

For most of the endpoints, Actuator offers two ways of connecting to them:

HTTP REST endpoint
JMX endpoint

By default, as we already saw, only health and info endpoints are exposed over HTTP. However, all of them are exposed over JMX (where applicable) by default.

You can configure in your application.properties which endpoints should be exposed:

# Expose over JMX
management.endpoints.jmx.exposure.exclude=shutdown
management.endpoints.jmx.exposure.include=*
# Expose HTTP REST Endpoint
management.endpoints.web.exposure.exclude=
management.endpoints.web.exposure.include=health,info,metrics

As you can see, you can define endpoints by name or use * to represent all endpoints. You can combine include and exclude filters.

If you change your configuration to expose all the endpoints over HTTP and restart your app, your /actuator endpoint will now show many more URLs.

management.endpoints.web.exposure.include=*

Enabling / Disabling endpoints

You can configure not only whether an endpoint is exposed over HTTP or JMX, but also you can turn specific endpoints on/off. All the endpoints except shutdown are enabled by default (although not exposed over HTTP).

# Disable an endpoint
management.endpoint.[endpoint-name].enabled=false

# Specific example for 'health' endpoint
management.endpoint.health.enabled=false

# Instead of enabled by default, you can change to mode
# where endpoints need to be explicitly enabled
management.endpoints.enabled-by-default=false

/health

As we already saw, the information provided by /health endpoint is rather unimpressive.

{"status":"UP"}

Fortunately, this is just the default state and we can show much more details.

management.endpoint.health.show-details=always

Now the information available is much richer. The exact details vary based on your specific dependencies. For example, whether you use a data source and which one.

{
  "status": "UP",
  "details": {
    "db": {
      "status": "UP",
      "details": {
        "database": "H2",
        "hello": 1
      }
    },
    "diskSpace": {
      "status": "UP",
      "details": {
        "total": 497336446976,
        "free": 202989006848,
        "threshold": 10485760
      }
    }
  }
}

Of course, it may not be a good idea to show such information to everyone publicly. Fortunately, you can restrict access, so the details are available to users with specific roles only. Instead of always, you rather use when-authorized and then you can specify the allowed roles.

management.endpoint.health.show-details=when-authorized
management.endpoint.health.roles=ADMIN

/info

When we tried to call /info endpoint before, all we've got was just an empty response {}. Of course, we can do better. Let's examine various ways we can display more info.

Build Properties

In the article I referenced at the beginning of this post, I describe how to obtain information about the artifact and its build properties. The idea is simple, configure Spring Boot Maven/Gradle plugin to generate build-info.properties file, which contains the required information.

What's great is that if you do use Actuator, it automatically detects build-info.properties file and displays its contents through the /info endpoint. All you need to do is to add a simple config to your Spring Boot Maven/Gradle Plugin.

Maven pom.xml file:

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <executions>        <execution>            <id>build-info</id>            <goals>                <goal>build-info</goal>            </goals>        </execution>    </executions></plugin>

Gradle build.gradle file:

springBoot {
    buildInfo()
}

Info endpoint now provides build info information:

{
  "build": {
    "version": "1.0.0-SNAPSHOT",
    "artifact": "spring-boot-actuator-example",
    "name": "spring-boot-actuator-example",
    "group": "com.vojtechruzicka",
    "time": "2018-09-01T19:03:06.446Z"
  }
}

Git Properties

Actuator automatically detects git.properties file, which contains useful information about your git repository. To generate it, you'll need to add a specific plugin to your build config.

In Maven pom.xml:

<plugin>
    <groupId>pl.project13.maven</groupId>
    <artifactId>git-commit-id-plugin</artifactId>
</plugin>

Gradle uses a different plugin:

plugins {
    id "com.gorylenko.gradle-git-properties" version "1.5.1"
}

After rebuilding and restarting the /info endpoint displays some git info.

{
  "git": {
    "commit": {
      "time": "2018-09-01T19:46:31Z",
      "id": "493f071"
    },
    "branch": "master"
  }
}

Environment properties

Finally, Actuator automatically detects all the environmental properties, which start with info.. You can try it by adding a new property to your application.properties, which starts with info..

info.my-custom-property=What a nice property!

Then it is returned by the /info endpoint.

{"my-custom-property":"What a nice property!"}

What's useful is that it shows properties from various sources, not only application.properties.

Environment variables
Command line arguments
Servlet Config/Context params
And many more

/metrics

This endpoint exposes information about various metrics. By calling /metrics you get names of all which are available.

{
  "names": [
    "jvm.memory.max",
    "jvm.memory.used",
    "jdbc.connections.active",
    "http.server.requests",
    "jvm.gc.memory.promoted",
    "system.cpu.usage",
    "tomcat.cache.hit",
    "tomcat.cache.access",
    "jvm.gc.max.data.size",
    "jdbc.connections.max",
    "jdbc.connections.min",
    "jvm.memory.committed",
    "system.cpu.count",
    "logback.events",
    ...
  ]
}

If you want to display data for a specific metric, instead of calling /metrics you call /metrics/{metric-name}. For example, calling /metrics/jvm.memory.used will return response similar to this:

{
  "name": "jvm.memory.used",
  "measurements": [
    {
      "statistic": "VALUE",
      "value": 2.44780736E8
    }
  ],
  "availableTags": [
    {
      "tag": "area",
      "values": [
        "heap",
        "nonheap"
      ]
    },
    {
      "tag": "id",
      "values": [
        "Compressed Class Space",
        "PS Survivor Space",
        "PS Old Gen",
        "Metaspace",
        "PS Eden Space",
        "Code Cache"
      ]
    }
  ]
}

Metrics were completely reworked in version 2. Version 1 uses its own proprietary metric system, which is hierarchical. This does not work very well in the cloud with many application instances. Actuator version 2 uses an entirely new system for managing metrics. It is dimensional in nature and it utilizes Micrometer.

Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.

So in addition to dimensional focus, you now have a nice facade to integrate various popular existing solutions such as Prometheus, Atlas, CloudWatch, Ganglia or New Relic. You can read more details in Micrometer: Spring Boot 2's new application metrics collector

Securing Actuator endpoints

Leaving your actuator endpoints exposed for anybody to access is not a good idea. That's why the vast majority of them are not exposed by default. Once you do expose them, you should make sure they are adequately protected.

The good news is that if you use Spring Security, Actuator endpoints are secured by default. If you're not using Spring Security, you can add this dependency:

Maven:
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>

Gradle:
compile 'org.springframework.boot:spring-boot-starter-security'

If you rebuild and restart your app, you'll notice that now you are required to log in if accessing the actuator endpoints. By default, the username is user and the password is randomly generated and printed to the console every time the application starts:

Using generated security password: f7b833aa-5a1c-42f4-b913-70c1abe47cb6

Protecting all the endpoints like this may be sufficient in many cases, you'll just usually change how user's credentials are validated. If you need more fine-grained control, you should create a configuration class extending WebSecurityConfigurerAdapter and override the configure method:

@Configuration
public class ActuatorSecurityConfiguration extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.authorizeRequests()
            .requestMatchers(EndpointRequest.to(ShutdownEndpoint.class))
                .hasRole("ADMIN")
            .requestMatchers(EndpointRequest.to(HealthEndpoint.class, InfoEndpoint.class))
                .permitAll()
            .requestMatchers(EndpointRequest.toAnyEndpoint())
                .fullyAuthenticated()
            .and().httpBasic();
    }
}

In the example above, the /shutdown endpoint is accessible only for an authenticated user with role ADMIN. Endpoints /health and /info are accessible for anybody. All the other endpoints are accessible only for fully authenticated users.

Notice that we are not matching by URL of the endpoints but rather by class, so we are not tightly coupled to the URL to which an endpoint is mapped to. If URLs are reconfigured, we don't need to change our security config.

Actuator 1.x vs 2.x

Many of the changes brought with Spring Boot 2.0 are in fact in the Actuator module.

Actuator endpoints are now technology independent. Before, they relied on Spring MVC. Now it does not matter whether you use MVC, Jersey or the new Spring WebFlux.

In version 1.x there was a special security configuration just for actuator endpoints. This was dropped and you now use Spring Security directly as you would for any other endpoint. It is much simpler and more consistent.

The old version of Actuator used to have proprietary metrics, but 2.x uses Micrometer. This was later backported even to 1.x.

There were some new endpoints introduced, some were renamed to more descriptive names. The configuration properties were also reworked and are now neatly grouped together by common prefix.

More detailed description of the changes can be found in Spring Boot 2.0 actuator change analysis. If you are migrating from version 1.x this guide may come in handy.

Adding Graphical User Interface

Actuator endpoints are great, but monitoring and managing your application just through JMX and HTTP endpoints may be too low-level and cumbersome for many. If you prefer a nice GUI instead of interacting with endpoints directly, there's a great tool just for this. It is called Spring Boot Admin. It is a third-party open-source project, which gives you a nice UI to manage and monitor your applications. What's more, a single instance of Spring Boot Admin can monitor multiple applications and/or multiple instances of each application. That's really useful in the cloud environment with many dynamic instances of your app.

Spring Boot Admin Tutorial

18 November, 2018

Monitor and manage you Spring Boot apps with a nice UI on top of Spring Boot Actuator endpoints. Works great with cloud and microservices.

Conclusion

Spring Boot Actuator offers a powerful solution for monitoring and managing your application in production. It offers interaction either over JMX or HTTP endpoints. If you prefer GUI instead try Spring Boot Admin on top of Actuator Endpoints. However, bear in mind that having public endpoints which allow you to tinker with your app and expose sensitive data is not a good idea. Always be sure to secure your endpoints.

Staticman: User generated content made static

Sat, 25 Aug 2018 12:12:03 GMT

Dynamic content in static sites

Static JAMStack sites are great. If you are not familiar with them, you can check my older post on how I migrated my blog from WordPress to a static site using Gatsby.

Migration to GatsbyJS and JAM stack from WordPress

25 March, 2018

How and Why I migrated from WordPress to static JAM Stack site built with Gatsby JS.

Long story short, you get excellent performance, increased security and more. Static sites do have their limitations though. Most of the pages use some kind of user-generated content. Usually, it is comments, user reviews, discussions, and various others.

With traditional static pages, this is somewhat problematic. With users submitting new content such as comments all the time, your site cannot be truly static. You usually end up with a solution such as Disqus. That is - a third-party service, which collects, manages and displays the user-generated content on your site. You have your core content static and then load a bunch of javascript, which fetches the dynamic content from third-party API. That's what A in JAMStack actually stands for - API.

This may be necessary for dynamic and ever-changing data such as stock prices, weather forecast and so on. However, comments or user reviews actually don't belong to this category. They are, in fact, static. Once you post a comment to a discussion it hardly ever changes. The discussion thread stays exactly the same until someone new adds a comment.

What if you could build your static site after a user comments including their new submission? Instead of loading the comment thread dynamically after the page loads, the whole thread could be core part of your static page. This way your site would be truly static with all the advantages.

Staticman

It turns out you can achieve this easily with a tool called Staticman. The idea is simple.

Provide a submission form to your static site. For example a form for submitting a new comment.
Since static sites have no backend, the form submits to a Staticman service instead.
Staticman pushes the changes to your git repo.
As usual, your site can be automatically rebuilt and redeployed on code change.
The user submitted data is now a core part of your static site.

How great is that? Of course, you don't want users spamming your site, so Staticman offers you an option to moderate submissions by creating a Pull Request instead of directly pushing to your branch. If you want more sophisticated protection, it does offer integration with Akismet anti-spam filter or re-captcha.

Setup

Granting Github access

The first thing you need to do is to grant Staticman privileges to access your Github repo. Go to Settings → Collaborators section of your repository and add staticmanapp as a collaborator.

After adding Staticman as a collaborator, you should see a pending invite:

Now you need to open the following URL in your browser to make Staticman accept your invite:

https://api.staticman.net/v2/connect/{your GitHub username}/{your repository name}

With a real data it looks something like this:

https://api.staticman.net/v2/connect/vojtechruz/staticman-example

If all goes well, you should receive response OK!. Now, in the collaborators section, you should no longer see the pending invite.

Adding a configuration file

Staticman loads its configuration from a yaml file called staticman.yml. A sample file can be found here. A detailed explanation of all the available configuration options can be found in the official documentation.

Here is a list of some of the most important configuration properties:

Property	Description
allowedFields	An array of the form fields, which should be accepted. If a field is received and it is not specified here, it is discarded.
branch	Name of the git branch Staticman should use.
commitMessage	Commit message to be used.
fileName	Each entry is stored as a file. This property determines the name. You can use placeholders such as @{id] or {@timestamp}.
format	Format of the files created. Can be yaml, json or frontmatter
moderation	If true, Staticman will create a Pull Request instead of pushing directly to your configured branch.
pah	Path to a directory where your Staticman files will be created.
requiredFields	Array of required fields. If some of them are missing, the entry will be rejected.

Creating a form

Now all you need is to create a form, which accepts user input and submits it to Staticman. You need to define all the fields which you want to collect and submit the form to:

https://api.staticman.net/v2/entry/{GITHUB USERNAME}/{GITHUB REPOSITORY}/{BRANCH}/{PROPERTY}

An example of such form:

<form method="POST" action="https://api.staticman.net/v2/entry/vojtechruz/staticman-example/master/comments">
  <label><input name="fields[name]" type="text" />Name</label>
  <label><input name="fields[email]" type="email" />E-mail</label>
  <label><textarea name="fields[message]"></textarea>Message</label>
  
  <button type="submit">Submit</button>
</form>

Note the /comments section at the end of action URL. This is a section identifier (marked as {PROPERTY} in the url template above), which allows you to have multiple configurations for your site. Eg. One for comments, one for user reviews etc. In this case, we're saying that we want to use configuration specified under the comments section of your configuration file.

comments:
  allowedFields: ["name", "email", "message"]
  ...
reviews:
  allowedFields: ["name", "email", "rating"]
  ...

If you want to redirect the user to a specific url after submit you can do it by including this hidden field:

<input name="options[redirect]" type="hidden" value="https://www.example.com" />

Integrating with your static site generator

Each user submission is stored as a separate file. You can define the location and format of these files in your configuration file.

Each file contains fields submitted by the user plus any automatically generated fields, which you can define in your config file.

_id: 749a3b20-a7a4-11e8-9d49-9fff9455a2a1
name: Vojtech Ruzicka
email: vojtech.ruz@gmail.com
message: Hello StaticMan!
date: 1535118541

What you need to do is to make sure your static site generator reads this data (you can set a different format than yaml if you need to) and generates static content based on it. This will, of course, vary greatly depending on which generator you use.

For example, if using Gatsby, you can use gatsby-transformer-yaml to read the yaml data and then just populate your comments section based on this data as you would with any other data source.

Moderation

You can turn on moderation of user submissions by setting moderation: true in your config file. Staticman then creates a Pull Request for each user submission, which you can either accept or decline.

Advanced features

Running your own instance

If using Staticman as a service does not suit your needs for any reason, you can run your own instance.

Akismet

Staticman offers integration with Akismet spam filter. This can be great when running your own instance, but unfortunately does not work currently on public Staticman instance as it is using shared Akismet account under the hood, which reached the maximum usage limit.

Email notifications

Staticman supports email notifications on new user submissions using Mailgun. You just need to enable the notifications in your config file and provide your Mailgun domain and API key.

Re-captcha

Staticman supports re-captcha integration. You'll need to create a re-captcha account first. You will receive a site-key and secret. Your secret is not used directly by Staticman and you need to encrypt it first.

https://api.staticman.net/v2/encrypt/SECRET

You should receive encrypted secret back in the response. You then use it in your staticman.yml config file:

reCaptcha:
  enabled: true
  siteKey: "SITE-KEY"
  secret: "ENCRYPTED-SECRET"

Then you need to include re-captcha javascript in your site:

<script src='https://www.google.com/recaptcha/api.js'></script>

Then the element where it should be rendered:

<div class="g-recaptcha" data-sitekey="YOUR-SITE-KEY"></div>

Finally you need to include these hidden fields in your Staticman form:

<input type="hidden" name="options[reCaptcha][siteKey]" value="SITE-KEY" />
<input type="hidden" name="options[reCaptcha][secret]" value="ENCRYPTED-SECRET" />

You can find detailed info about re-captcha integration here.

Example

The source code of a simple GatsbyJS site using Staticman can be found here.

Running example is available here.

Conclusion

Staticman is an excellent tool if you want to have user-generated content on your site while remaining truly static. You no longer need to rely on third-party services, to manage and store your content. There is no need to load the user-generated content from third-party API at runtime. This should give you better performance and you have direct control over your data. If you need moderation of submitted content, Staticman can create Pull Requests, so you can decide which submissions are acceptable and which not.

Detecting build version and time at runtime in Spring Boot

Mon, 25 Jun 2018 22:12:03 GMT

This post covers obtaining simple build-related information without adding any additional dependencies. For much more detailed info, various metrics and health monitoring you should consider using Spring Boot Actuator:

Actuator: Spring Boot Production Monitoring and Management

03 September, 2018

Monitor and manage your application in production with Spring Boot Actuator 2.x. Gather metrics or check health easily.

Obtaining build information

It can often be useful to obtain information about artifact, version, build time and other at runtime. Sure, most of this information is already in your pom.xml file, but it can be tricky to retrieve these when the application is running.

Having such information at runtime can be useful. For example, imagine a scenario, where you expose a REST endpoint, which can tell the client what your current version of the application is, when was it built and so on. It can be useful because you can quickly determine what version of the app is currently deployed. This can be especially important in non-production environments, where the app is frequently deployed or even with continuous deployment in production. In such cases, it is vital to know what build exactly is currently running when testing and submitting bug reports. Maybe the issue reported is already fixed in a newer version or maybe the bug still occurs because the new version is implemented, but not deployed yet.

In any case, having build information can be handy and it is useful to know how to obtain it at runtime. In Spring Boot, it is fortunately quite easy.

Build plugin configuration

If you are using Spring Boot, your pom.xml should already contain spring-boot-maven-plugin. You just need to add the following configuration.

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <executions>        <execution>            <id>build-info</id>            <goals>                <goal>build-info</goal>            </goals>        </execution>    </executions></plugin>

It instructs the plugin to execute also build-info goal, which is not run by default. This generates build meta-data about your application, which includes artifact version, build time and more.

If you are using Gradle, just add the following to your build.gradle file:

springBoot {
    buildInfo()
}

Accessing Build Properties

After configuring your spring-boot-maven-plugin and building your application, you can access information about your application's build through BuildProperties object. Let the Spring inject it for you:

@Autowired
BuildProperties buildProperties;

Now you can access various information from this object.

// Artifact's name from the pom.xml file
buildProperties.getName();
// Artifact version
buildProperties.getVersion();
// Date and Time of the build
buildProperties.getTime();
// Artifact ID from the pom file
buildProperties.getArtifact();
// Group ID from the pom file
buildProperties.getGroup();

Adding custom properties

If predefined properties are not enough, you can pass your own properties from pom.xml file to BuildProperties.

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <executions>
        <execution>
            <goals>
                <goal>build-info</goal>
            </goals>
            <configuration>                <additionalProperties>                    <java.version>${java.version}</java.version>                    <some.custom.property>some value</some.custom.property>                </additionalProperties>            </configuration>        </execution>
    </executions>
</plugin>

You can pass a value directly or use your custom properties defined in the <properties> section of your pom.xml and then referenced using ${property.name} placeholder.

You can access custom properties defined this way by calling buildProperties.get("property.name").

For Gradle projects, custom properties can be defined this way:

springBoot {
    buildInfo {
        properties {
            additional = [
                'property.name': 'property value',
                'other.property': 'different.value'
            ]
        }
    }
}

How it works under the hood

When build-info of spring-boot-maven-plugin is run, it generates a property file containing all the build information. By default, it is located at ${project.build.outputDirectory}/META-INF/build-info.properties, but you can customize it by providing outputFile parameter. The file looks something like this:

#Properties
#Sat Jun 23 15:58:56 CEST 2018
build.version=0.0.1-SNAPSHOT
build.group=com.vojtechruzicka
build.name=spring-rest-docs-example
build.artifact=spring-rest-docs-example
build.time=2018-06-23T13\:58\:56.742472800Z

When Spring detects there is this file on the classpath, it creates BuildProperties bean unless it is explicitly declared. This is configured in org.springframework.boot.autoconfigure.info.ProjectInfoAutoConfiguration. This is a nice example of Spring Boot Auto-Configuration, where certain beans can be created just by having specific files on the classpath.

@ConditionalOnResource(resources = 
"${spring.info.build.location:classpath:META-INF/build-info.properties}")
@ConditionalOnMissingBean
@Bean
public BuildProperties buildProperties() throws Exception {
    return new BuildProperties(
            loadFrom(this.properties.getBuild().getLocation(), 
            "build"));
}

Detecting Spring profiles

It is no doubt useful to know which version of your artifact is deployed and when it was built. However, it is usually not enough. Often Spring applications use various profiles, which can significantly change the behavior. Typical usage is, for example, having a separate profile for each environment (DEV, UAT, PROD, ...). Depending on the profile, the correct environmental configuration can be loaded such as DB connection and more.

It is useful to be able to determine current profiles as sometimes the app can be run with different profiles than expected. To detect the current profiles, you need just to inject Environment object and then you can simply obtain them by calling getActiveProfiles().

@Autowired
private Environment environment;

environment.getActiveProfiles();

What's more, since you already have environment object, you can obtain any environmental properties by calling environment.getProperty("property.name").

Spring Actuator & Admin

While this approach gives you basic build and version info, sometimes you may need a more powerful tool. Spring Boot Actuator is a sub-project of Spring Boot, which adds some production-grade monitoring and management tools exposed as REST and JMX endpoints. In fact, it can easily be configured to use build information provided by BuildProperties and provide them through one of its endpoints.

Spring Boot Admin is a community project, which provides a nice user interface on top of Spring Actuator endpoints, so the app is more comfortable to manage through a nice admin GUI.

Conclusion

Having access to version and build information at runtime can be quite useful. In Spring boot application, you can easily obtain the info by altering the Spring Boot Maven/Gradle plugin configuration to generate the build.properties file and then accessing it through BuildProperties object.

For simple scenarios, this is an easy and quick solution and should work for you. If you need something more powerful, look at Spring Actuator or Spring Admin, which can provide the build metadata functionality plus a lot more.

Easy git hooks with Husky

Thu, 21 Jun 2018 22:12:03 GMT

In this article, you will learn how to easily create and manage git hooks for your Node/NPM projects using Husky.

Git Hooks

Simply put, git hooks are custom scripts, which can be run automatically when specific events occur. There are client-side hooks which are triggered on actions such as committing or merging. Server-side hooks run in situations such as receiving push data from the client.

The hooks can perform any custom logic and most importantly reject the action performed if something is not in order. For example, you can abort commit if its message does not contain issue tracker's issue ID. Or, you can reject it if static code analysis fails. It can be pretty useful if you want to make sure your codebase stays clean or you want to enforce certain quality policies.

But how can you actually install and manage these hooks? Whenever you clone a git repository, all the git's data for your project is stored in .git directory inside your folder. It contains several files and sub-directories, one of which is called hooks. Inside you'll find a bunch of files.

Each of them is a script, which is executed once a specific event occurs. The name of the event matches name of the file. So for example pre-commit is executed before you commit your changes.

As you can see, all the files have extension .sample. Git ignores these files unless you rename them - you need to remove the .sample extension to enable these hooks. Inside the sample files, you can find some description and sample implementation, which you can use as a starting point when implementing your own hooks.

What is it good for?

Now, let's look at some specific examples of what can be achieved with git hooks. Since they're just scripts, you can do pretty much anything. Usually, that means making various quality checks.

You can make sure user has their name and email filled. You can check the commit message is properly formatted. You can try to build your app and reject the commit if the build fails. You can run tests to make sure they are passing before commit.

Typical usage is also static code analysis or linting. That means checking your code for common issues, bad practices, naming conventions and more. It can also be useful to run a tool such as Prettier to make sure the code is nicely formatted before commit. This saves many headaches during code reviews. You can even check your code for security vulnerabilities with a tool like Snyk.

Snyk – Detecting dependencies with known vulnerabilities

22 November, 2017

#Security

How to detect and fix security vulnerabilities in your dependencies using Snyk?

Distribution to team members

With server-side hooks the distribution is easy. You usually have just one primary server repository. This means all the team members usually push their changes to one central place. You set up your hooks there and you're done.

With client-side hooks, this gets more complicated. When you clone a repository, the hooks are not transferred to the client. That means a freshly cloned repo does not have any hooks at all no matter what hooks you have server-side. If you want your team members to have unified set of git hooks, you need to distribute them somehow and make sure they are included in their git hooks sub-directory.

The most basic solution is to have some shared location, where you store your hooks and then tell your developers to download them and put them in their hooks directory. Of course, you cannot be sure that they will actually do it. The problem is they need to know they should do this and how. And even if they do, they may be lazy or just ignore your policy.

You can improve this basic solution a bit by having hooks in your project's repo and just let your developers run a custom script like this one, which copies them over to their hooks directory. Alternatively, git offers an option to change the destination of the hooks directory to a custom location:

git config core.hooksPath YOUR_DIRECTORY

These solutions make the distribution easier, but don't solve the core issues.

Husky

Installation

You can install Husky simply by running:

npm install husky --save-dev

Alternatively, with yarn:

yarn add husky --dev

Adding hooks

Adding hooks with Husky is easy. You just need to edit your package.json. There you define which scripts should be run on which git event.

For Husky 1.0.0+ use:

{
  "husky": {
    "hooks": {
      "pre-commit": "npm test",
      "pre-push": "npm test",
      "...": "..."
    }
  }
}

If you are using a Husky version older than 1.0.0 (which is in Release Candidate as of 6/2018), the syntax is slightly different. You add your Husky hooks directly in your scripts section.

{
  "scripts": {
    "precommit": "npm test",
    "prepush": "npm test",
    "...": "..."
  }
}

The example above runs all your tests before you commit and before you push and if your tests fail the git action is not executed. Of course, you can run any other npm script rather than npm test.

Autofix issues before commit

It is useful to break the build if something is not right, but it is even more useful to auto-correct issues before committing. For example, you can prettify your code using Prettier before commit or you can auto-fix lint issues, which can be resolved automatically. It is much easier this way.

Fortunately, there is a tool exactly for this. It is called lint-staged. You can install it by:

npm install --save-dev lint-staged

Now on your pre-commit action, you execute lint-staged instead of your linter directly. In lint-staged configuration you define what should be executed:

{
"husky": {
    "hooks": {
      "pre-commit": "lint-staged"
    }
  },
 "lint-staged": {
   "*.js": ["eslint --fix", "git add"]
 }
}

When you try to commit now, lint-staged can change your files before the commit is executed. However, what's really cool is that you are linting only files staged to commit, not your whole project. That means the whole process is way faster.

JetBrains IDE bug

UPDATE: The bug described below is fixed in version 2018.3, if you are using an older version you can use this workaround:

"post-commit": "git update-index -g"

The good news is that git hooks configured in this way are executed not only when you use git from your command line, but also from an IDE. The bad news is that JetBrains IDEs (IDEA, Webstorm, ...) have currently a nasty bug (see IDEA-135454) and doesn't work well with this setup.

~~The issue is not resolved for several years, but fortunately, there is a workaround. You just need to add this post-commit hook:~~

~~"post-commit": "git update-index -g"~~

~~Of course, this is just workaround until the issue is fixed. JetBrains's issue tracker contains voting functionality, so be sure to vote for this issue to be resolved if it causes you trouble.~~

Continuous integration

One thing to note is that Husky installs the hooks only when not running on a Continuous Integration server. Husky can detect it is running as a part of CI job and will not install any hooks.

Ignoring

Client-side hooks can be useful, but you cannot rely on them too much. They are just the first level of defense. You cannot be 100% sure that they get executed. They can be ignored on demand by adding a command-line option:

git commit --no-verify

To make things even easier, the hooks can be disabled using certain environmental variables. Because of this, it is still useful to enforce the same functionality on the server.

Performance

While client-side hooks such as pre-commit can be very useful, you need to keep in mind that they take some time to execute. Commits, which are usually very fast as they happen only on the client, can suddenly take very long time. You may be tempted to run all the tests, static code analysis, prettification and more before each commit. When a commit takes ages, your developers will not be happy and may be tempted to ignore the hooks when performing their git commands. So you should find the right balance between what needs to be performed on the client and what can be a server-side hook.

Conclusion

Husky is a useful tool, which allows effortless creation and management of git hooks on the client. You no longer need to distribute your hooks manually. As with everything, keep the number of your client-side hooks in moderation to avoid long execution times.

Adding search to your static JAMStack site

Sun, 17 Jun 2018 22:12:03 GMT

Searching static sites

A few months ago I finally managed to migrate my blog from WordPress to a static site build with GatsbyJS and deployed on Netlify.

Migration to GatsbyJS and JAM stack from WordPress

25 March, 2018

How and Why I migrated from WordPress to static JAM Stack site built with Gatsby JS.

I was very impressed with the results of the migration, especially performance-wise. I was able to migrate all the functionality, except site-wide search.

With server generating your pages, searching is easy. WordPress offers search option out of the box. With static sites, which have no dynamic backend, this gets trickier.

Client-side search

One option to enable search for your application is to use full client-side search with libraries such as js-search. The idea is simple. When you build your static site using your favorite static site generator, you also generate a search index of your pages. That basically means generating a lot of JSON with the contents of your site and then distributing this along with your static files. The search is lightning-fast because everything happens on the client. The problem is that this solution may not be good for full-text search. Bringing this much data to the client just for search can mean a significant performance hit. Especially if your site has a lot of content. Moreover, just a fraction of users is likely to use the search option. The rest just download useless junk. You sacrifice one of the greatest advantages of going static - performance just for adding a search option.

Good old Google Custom Search

All right, so for full-text search, it may not be such a good idea to download the whole search index to the client. Since our static site does not have a dedicated backend, we need to use third-party to handle the search. The first obvious choice is to use Google search. Google Custom Search to be precise.

The advantage is it doesn't need to download the whole search index. You use Google to search your site only. There are some disadvantages though. First one is that Google injects ads to your search results. The second one is that even though you can customize the look and feel of the search results page to some degree, it is still obvious that you utilize Google. The results page will likely not fit very well the look and feel of your page and arguably it does not look very well integrated and professional. Especially with the ads. Like you didn't care enough to provide custom search as other sites do.

Alternatively, if you don't mind doing some extra work, custom search offers an API, so you can retrieve results programmatically and seamlessly integrate them in your page. This, however, somewhat negates the most significant advantage of Google Custom search, which is its easy setup.

If you don't care about the downsides much, this can be very easy and quick way to add search for your static site. If you don't like it, there are still alternatives.

Algolia DocSearch

Algolia DocSearch is similar to Google Custom Search in a way that you don't need to download the whole search index to the client and that the search is provided by third-party service.

There are some advantages over Google Search though. First of all, no ads. Also, everything integrates with the look and feel of your site nicely as you just mark a text input on your page as a search box to be handled by DocSearch. It then attaches autocomplete functionality to this search box. Whenever you start typing you get results from DocSearch. Directly in your page with possibility to easily override default styles. So the setup is really easy and your pages are indexed for you automatically.

Requesting access

The first step you need to make is to request DocSearch for your site. Just fill a simple form on their website.

If your request gets approved, you're good to go. They will configure the crawler for your site, which will gather search data once a day. And they will send you JS and CSS to include on your page to enable the search. That's it.

But your request may get rejected (as in my case) because DocSearch is primarily intended as a solution for searching documentation pages of open source projects.

However, after the initial rejection, I did ask nicely whether they can provide the service for my blog. They were really kind to me and allowed me to register.

DocSearch focuses on documentation content that helps the community. It looks like your content is helpful this why we will provide you DocSearch.

DocSearch on your own infrastructure

However, even if your request is rejected, nothing is lost. The whole DocSearch is open-source and you can install and run it on your own infrastructure. It's a viable alternative as the search metadata is still managed and stored externally and collected by DocSearch crawler.

Setup and configuration

One of the advantages of DocSearch how easy it is to set up and configure. You just need to add the following to your page (Algolia team provides you with exact values and details after the registration):

<!-- at the end of the HEAD --> 
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css" /> 

<!-- at the end of the BODY --> 
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js"></script> 
<script type="text/javascript"> docsearch({ 
 apiKey: '### PROVIDED  BY ALGOLIA ####', 
 indexName: '### PROVIDED  BY ALGOLIA ####', 
 inputSelector: '### REPLACE ME ####', 
 debug: false // Set debug to true if you want to inspect the dropdown 
}); 
</script>

You just need to provide proper selector (such as CSS class or id) for the input search element. Algolia team also configures the project for you and they store the configuration file in their git repo. In case you need to make any changes, you just create a Pull Request, which gets quickly merged. In the config file, you define a hierarchy of your pages sections. That means, for example, that H1 headings have more weight in search results than H2. And H3 have even less weight. Ordinary paragraphs have the least relevance. With custom selectors you can define where precisely in your page is your content, so you can easily ignore sections such as footer or navigation. Here is an example of a simple configuration for my site:

{
  "index_name": "vojtechruzicka",
  "start_urls": [
    "https://www.vojtechruzicka.com"
  ],
  "stop_urls": [],
  "selectors": {
    "lvl0": {
      "selector": "",
      "global": true,
        "default_value": "Search Results"
    },
    "lvl1": ".docSearch-content h1",
    "lvl2": ".docSearch-content h2",
    "lvl3": ".docSearch-content h3",
    "lvl4": ".docSearch-content h4",
    "text": ".docSearch-content p, .docSearch-content li"
  },
  "conversation_id": [
    "593663647"
  ],
  "nb_hits": 1890
}

If you're looking for a more complex example, you can check configuration of GatsbyJs.

Algolia Search

DocSearch is primarily focused on documentation and full-text search. However sometimes you need something more powerful and sophisticated. In such case, Algolia's flagship product - Algolia Search is better for you.

Instead of working with the full text of your pages, you can define your own structured data in the form of JSON objects and Algolia works with them. Each of these objects can have many properties and you can search across all of them.

On top of this Algolia offers many advanced features, which are not available in DocSearch. You can define how the search results should be ranked, so the user receives the most relevant results. You can define synonyms for search terms. You can provide correction alternatives to common typos. There is a nice dashboard and search logs. You can even have analytics with a paid plan.

Uploading search data

Algolia does not crawl your content automatically as DocSearch does. You need to upload your data to their servers using REST API. They offer clients for a number of languages and frameworks such as Java, C#, PHP, Python, Django, Laravel, Rails or Symfony.

While this may be useful for traditional applications, it is not of much use for static sites. In this case, you need to build your JSON objects as a part of building your static site and then upload them using their REST API.

Of course, you can do it manually and it will work. With some static site generators, it is your only option. Fortunately, some of the generators provide plugins for Algolia, which make the process much easier. For example there is a plugin for Jekyll or GatsbyJs (as of 6/2018 in beta).

Search your videos

If you also work with videos, Algolia offers another exciting service. It is called Algolia TalkSeach. It allows you to add search to your youtube videos. It fetches transcript for the video and makes it searchable, so you can easily navigate to the part of the video you are interested in.

It is intended primarily for conferences and meetups, but again, they may be kind enough to make an exception for you.

Conclusion

When adding search to static sites, you need a different approach than in traditional apps where your backend can handle the search for you. Adding Google Search can be quick and painless, but the customization is limited and it does not feel like a nice integrated solution.

Having client-side search offers a fast solution, which can be good if you want your app to work offline as a Progressive Web App. Of course, you need to download the whole search index to the client, which can be too much once you have a lot of content.

Algolia DocSearch is an easy to set up alternative, which crawls your site for you. So the data is stored on the server, so you don't have to download everything.

If you need more robust approach and more control, regular Algolia may be a better solution, but the setup is somewhat more complicated as you need to upload your data when you make changes to your site.

Spring REST Docs - Test driven documentation of your REST API

Thu, 31 May 2018 22:12:03 GMT

SpringFox And Swagger

Traditional and popular approach to documenting your REST API is Swagger (aka OpenAPI), which I covered in detail in one of my previous articles.

Documenting Spring Boot REST API with Swagger and SpringFox

16 February, 2018

How to document your Spring Boot REST APIs using Swagger with SpringFox?

It is a powerful tool, which can be useful especially when you want to generate your docs for your existing API effortlessly. It detects all the endpoints, input and output parameters automatically and generates the docs. However, if you want to provide custom descriptions of all the endpoints and fields, you'll end up with polluting your code with many annotations. The vast majority of your Controller and Model classes' code is plagued with documentation-specific annotations, which makes them hard to read. In the example below, everything except the highlighted lines is REST API documentation specific.

@ApiOperation("Creates a new person.")
@ApiResponses(value = {
    @ApiResponse(code = 200, message = "Success", response = Person.class),
    @ApiResponse(code = 401, message = "Unauthorized"),
    @ApiResponse(code = 403, message = "Forbidden"),
    @ApiResponse(code = 404, message = "Not Found"),
    @ApiResponse(code = 500, message = "Failure")})
@RequestMapping(method = RequestMethod.POST, produces = "application/json")    public Person createPerson(        @ApiParam("Person information for a new person to be created.")
        @RequestBody Person person) {    return personService.createPerson(person);}

Spring REST Docs

Spring REST Docs take a different approach. Instead of infesting your controller and model classes with documentation annotations, it moves all this information elsewhere. To your tests. To be precise - to the tests of your Controllers. It is convenient because for many people tests are the best place to look at when trying to understand how some functionality works. Because unlike comments and documentation, tests are always up to date. When your tests are out of date and no longer in sync with your implementation, they start to fail.

This is important because if your documentation is outdated, people no longer trust it and it becomes useless. Having outdated documentation is worse than having no documentation at all. That brings us to what's so cool about Spring Rest Docs. It is tightly integrated with your tests. So when your documentation gets different from your implementation, your tests start to fail. For example, if you add a field, which is not documented, your test no longer passes. If you remove a field and it is still in your documentation, the test also fails.

Spring Test MVC

For the behavior described above to work, Spring Rest Docs need to be integrated with your test framework, which you use to test your REST API. There are various options in Spring, but Rest Docs currently support Spring MVC Test, Spring Webflux's WebTestClient and RestAssured. In this tutorial, I'll cover Spring MVC Test, but you can use any of them.

Setting Up Spring Rest docs

Starting repository

To follow this tutorial, you can use any Spring/Spring Boot application with REST controllers. You can either use your own or build on top of a sample starter repository I prepared for this purpose.

The final source code for this tutorial is available in this repository.

Creating Spring MVC Tests

First, before diving deep into Spring Rest Docs specifics, you'll need some regular tests of your controllers. Let's create a simple one, which calls a controller's method and checks whether HTTP response code is 200 OK and the content type of the response is JSON. Of course, you can test much more, such as the response data, HTTP headers, cookies and so on.

package com.vojtechruzicka.springrestdocs.controllers;

...

import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.content;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

@RunWith(SpringRunner.class)
@SpringBootTest
public class PersonControllerTest {

    @Autowired
    private WebApplicationContext context;

    private MockMvc mockMvc;

    @Before
    public void setup() {
        mockMvc = MockMvcBuilders.webAppContextSetup(context).build();
    }

     @Test
     public void getPersonByIdShouldReturnOk() throws Exception {
         this.mockMvc.perform(MockMvcRequestBuilders.get("/persons/1"))
                 .andExpect(status().isOk())
                 .andExpect(content().contentType("application/json;charset=UTF-8"));
     }

}

So far, nothing REST Docs specific, but this is the base we'll be building on.

Adding the dependencies

The first thing you'll need to do is provide the Spring Rest Docs dependency. You'll need to use a different one depending on whether you want to use Spring MVC Test, WebTestClient or RestAssured. For Spring MVC Test use the following (the latest version).

<dependency> 
    <groupId>org.springframework.restdocs</groupId>
    <artifactId>spring-restdocs-mockmvc</artifactId>
    <version>2.0.1.RELEASE</version>
    <scope>test</scope>
</dependency>

Or for Gradle:

testCompile 'org.springframework.restdocs:spring-restdocs-mockmvc:2.0.1.RELEASE'

Configuring your tests - Junit 4

Let's add a specific @Rule for REST documentation and then use it when building the mockMvc object. Only the highlighted lines below are new. The rest is the original code sample we already saw.

@RunWith(SpringRunner.class)
@SpringBootTest
public class PersonControllerJunit4Test {

    @Autowired
    private WebApplicationContext context;

    private MockMvc mockMvc;

    @Rule    public JUnitRestDocumentation jUnitRestDocumentation = new JUnitRestDocumentation();
    @Before
    public void setup() {
        mockMvc = MockMvcBuilders
                .webAppContextSetup(context)
                .apply(documentationConfiguration(this.jUnitRestDocumentation))                .build();
    }

    ...
}

Configuring your tests - Junit 5

For JUnit 5, the configuration is also easy. You need to use RestDocumentationExtension.class extension in addition to Spring's one you would use normally. Then when constructing the mockMvc object apply the configuration. You're adding just the highlighted lines in the example below.

@SpringBootTest
@ExtendWith({ RestDocumentationExtension.class, SpringExtension.class})public class PersonControllerJunit5Test {

    private MockMvc mockMvc;

    @BeforeEach
    public void setUp(WebApplicationContext webApplicationContext,
                      RestDocumentationContextProvider restDocumentation) {        this.mockMvc = MockMvcBuilders
                .webAppContextSetup(webApplicationContext)
                .apply(documentationConfiguration(restDocumentation))                .build();
    }

    ...
}

Generating the Documentation

Now when we have the test configuration ready, it's time to write some documentation. First, let's provide a command to generate the documentation in each test method. Just add andDo(document("[documentation snippet's name]")). Then you need to replace MockMvcRequestBuilders with RestDocumentationRequestBuilders.

    @Test
    public void getPersonByIdShouldReturnOk() throws Exception {
        this.mockMvc.perform(
                RestDocumentationRequestBuilders                .get("/persons/1"))
                .andExpect(status().isOk())
                .andExpect(content().contentType("application/json;charset=UTF-8"))
                .andDo(document("persons/get-by-id"));    }

Now let's run the tests to make sure everything works fine. They should pass and in your target/generated-snippets should be a sub-folder with a name matching the string you provided in andDo(document("[documentation snippet's name]")). Inside, there should be a bunch of .adoc files.

If you don't want to provide snippet's name explicitly, you can provide placeholders, such as:

andDo(document("{ClassName}/{methodName}")

AsciiDoc vs. Markdown

The files inside are fragments of your documentation. They contain information such as HTTP request and response or curl commands to call your endpoints.

The resulting API documentation should be, of course, HTML. However, as you can see, it is in a different format now. It's called AsciiDoc and it is very similar to MarkDown. That means it's a simple markup language for text formatting written in plain text. MarkDown is widespread, well-known language, so why to introduce another one? Why not stick with MarkDown?

Why You Shouldn’t Use Markdown for Documentation sums the reasons pretty well.

Because the original Markdown is so limited, every popular tool built on top of Markdown implements what is called a “flavor” of Markdown. This sounds great, except that every tool implements a different flavor. Even tools that do similar things with the language use different syntax for it! ... In the last few years, Commonmark was developed as a standardized Markdown. This is great, and should solve lots of problems! Except that nobody has adopted it...

You can read more about the differences in this comparison.

Converting AsciiDoc

Since AsciiDoc cannot be directly rendered by a browser, we need a way to convert the documentation from AsciiDoc to HTML. There's a tool called AsciiDoctor, which is also available as a Maven plugin (there's also one for Gradle). Just include the following in your pom.xml file.

<build>
    <plugins>
        <plugin> 
            <groupId>org.asciidoctor</groupId>
            <artifactId>asciidoctor-maven-plugin</artifactId>
            <version>1.5.5</version>
            <executions>
                <execution>
                    <id>generate-docs</id>
                    <phase>prepare-package</phase> 
                    <goals>
                        <goal>process-asciidoc</goal>
                    </goals>
                    <configuration>
                        <backend>html</backend>
                        <doctype>book</doctype>
                    </configuration>
                </execution>
            </executions>
            <dependencies>
                <dependency> 
                    <groupId>org.springframework.restdocs</groupId>
                    <artifactId>spring-restdocs-asciidoctor</artifactId>
                    <version>2.0.1.RELEASE</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

By default, this plugin does not run in any of the Maven lifecycle phases. So you need to bind it to one. Prepare-package is convenient if you want your generated documentation to be included in the resulting package and for example being served by Spring Boot as static resources.

Putting the snippets together

Having .adoc snippets and AsciiDoctor Maven plugin is still not enough. You need to provide more AsciiDoc files, which will 'glue' together your generated snippets. In these files you can put any additional documentation and description needed by your users and you can choose which snippets to include and which not.

This is one of the advantages over the good old Swagger. It is not just documentation of your API endpoints, but you can include huge chunks of additional documentation or even whole pages. This means you can combine traditional documentation with API docs.

What you need to do is to create a directory src/main/asciidoc and create your new .adoc file there.

When you run mvn package, all .adoc files from src/main/asciidoc will be converted to HTML and copied to target/generated-docs. Lets create a file called src/main/asciidoc/index.adoc.

= Sample API Documentation

== Introduction
This is an example of Spring REST Docs generated documentation.

== Persons API
Collection of CRUD API endpoints used to manipulate persons registered in the application.

=== Get Person By Id
Obtains a specific person registered in the application by its unique identifier.

==== Sample Request
include::{snippets}/persons/get-by-id/http-request.adoc[]

==== Sample Response
include::{snippets}/persons/get-by-id/http-response.adoc[]

==== CURL sample
include::{snippets}/persons/get-by-id/curl-request.adoc[]

As you can see, you can divide the document to any number of sections and provide any text you want. When you want to use previously generated snippets you use include statement like include::{snippets}/persons/get-by-id/curl-request.adoc[]. Now just run the maven build to make sure this AsciiDoc file is converted to HTML.

mvn clean package

The resulting file should be available under target/generated-docs/index.html and it should look like this:

So far so good. Now let's tweak it a bit more. Let's add information about the author and version, the table of contents to be displayed on the left side and finally an automatic numbering of section headings. Just include the highlighted lines below the document heading.

= Sample API Documentation
Vojtech Ruzicka<myfakemail@gmail.com>1.0.0, 30/5/2018:toc: left:sectnums:

To learn more about writing AsciiDoc, check the user's manual.

Documenting path params

We generated some documentation already, but so far it contains just a sample request and response and not much more. There's still much to be documented though.

Let's document the following endpoint more:

    @RequestMapping(method = RequestMethod.GET, path = "/{id}", produces = "application/json")
    public Person getPersonById(@PathVariable int id) {
        return personService.getPersonById(id);
    }

Let's start with the PathVariable. That means that id of the person is passed in the URL as /persons/{id}.

@Test
public void getPersonByIdShouldReturnOk() throws Exception {
    this.mockMvc.perform(RestDocumentationRequestBuilders.get("/persons/{id}", 1))
        .andExpect(status().isOk())
        .andExpect(content().contentType("application/json;charset=UTF-8"))
        .andDo(document("persons/get-by-id",
               pathParameters(parameterWithName("id").               description("Identifier of the person to be obtained."))));
}

When you run maven build again, your HTML document does not change. The reason is that you just generated a new snippet called path-parameters.adoc, however, you still need to include it in your index.adoc file. Let's add a new section then:

==== Path Parameters
include::{snippets}/persons/get-by-id/path-parameters.adoc[]

If you build again, you should see a new section about path params in your documentation.

Documenting request and response payload

Our controller's method getPersonById() returns a person represented as JSON.

{
  "id": 1,
  "firstName": "John",
  "lastName": "Smith",
  "age": 42
}

Let's document all the fields using responseFields() metod:

@Test
public void getPersonByIdShouldReturnOk() throws Exception {
    this.mockMvc.perform(RestDocumentationRequestBuilders.get("/persons/{id}", 1))
        .andExpect(status().isOk())
        .andExpect(content().contentType("application/json;charset=UTF-8"))
        .andDo(document("persons/get-by-id",
            pathParameters(parameterWithName("id")
                .description("Identifier of the person to be obtained.")),
            responseFields(                fieldWithPath("id")                    .description("Unique identifier of the person."),                fieldWithPath("firstName")                    .description("First Name of the person."),                fieldWithPath("lastName")                    .description("Last Name of the person."))        ));
}

Now when you rerun the test, it fails horribly:

org.springframework.restdocs.snippet.SnippetException: 
The following parts of the payload were not documented:
{
  "age" : 42
}

This is one of the most significant features of Spring Rest Docs in action. We forgot to include one of the response fields in the documentation, but the API still returns it. In other words, there is an undocumented field. If this happens, your tests start to fail, so your docs are always up to date. Moreover, it works the other way around too. If you document a field and your controller no longer returns it, your tests start to fail. It's really powerful.

If you add documentation also for the age field and build again, it generates yet another snippet - response-fields.adoc. You know the drill - include it in your docs and build again. And voila!

You can see Spring also inferred data type of the fields without us specifying it explicitly.

Documenting the request payload is pretty much same. Just use requestFields() method and request-fields.adoc fragment.

Documenting request parameters

Request parameters can be passed in two ways. First, you can add them after question mark at the end of an URL:

this.mockMvc.perform(RestDocumentationRequestBuilders
    .get("/persons?limit=100&order=asc"))

Or in a body of a POST request:

this.mockMvc.perform(RestDocumentationRequestBuilders
    .post("/persons")
    .param("limit", "100")
    .param("order", "asc"))

Either way, documenting request params is the same:

.andDo(document("persons/get-all", requestParameters(
    parameterWithName("limit")
        .description("Limit the number of persons obtained to this number."),
    parameterWithName("order")
        .description("Orders persons by name alphabetically in either ascending (asc) or descending (desc) order.")
)));

The resulting snippet is request-parameters.adoc.

Spring Auto Rest Docs

There is an interesting alternative to vanilla Spring Rest Docs called Spring Auto Rest Docs. Its aim is to automate the process of documentation as much as possible. Instead of documenting all the JSON fields manually, it can generate the docs for them automatically by introspecting your classes, reading field names, types and the description from JavaDoc.

Generating WireMock stubs from your API docs

Even more interesting is a project offering Spring REST Docs WireMock Integration. It can generate WireMock stubs from your Spring Rest Docs API Documentation. This is really powerful as it ensures both your API documentation AND your API stubs are always up to date. Pretty cool, right? You can read more in this blog post.

Conclusion

While Swagger and SpringFox are not a bad choice, Spring Rest Docs offers some powerful benefits you should definitely consider. The main one is having your docs always up to date, because if the documentation goes out of sync, the tests start to fail. Another one is that it motivates you to write tests of your controllers as they are required to create your API documentation.

Also, with Spring Rest Docs it is much easier to write documentation not only of your API directly, but also add any other necessary info such as tables, blocks of text, images, code blocks and so on. And of course, your production code is no longer plagued by all the documentation annotations, which make it so hard to read.

Swagger, on the other hand, offers a compelling option to call your API directly from the documentation. Also, it effortlessly generates basic information about your API without you writing anything. It can be useful for a legacy codebase to have at least some documentation in no time. Of course, without custom descriptions and explanations, but at least it is something.

Migration to JAM stack and Netlify from WordPress

Mon, 21 May 2018 22:12:03 GMT

JAM Stack Migration - Part 2

Recently I managed to finally get rid of WordPress and migrate my blog to JAM Stack, built with a static site generator called GatsbyJS. I described what JAM Stack is, how I migrated and how great Gatsby is in my previous post - Migration to GatsbyJS and JAM stack from WordPress.

This second post describes build and deployment process and how much it is better than the original WordPress setup or traditional GitHub Pages.

GitHub Pages?

Leaving managed WordPress saved me some expenses, but the next question was where to deploy instead. The first obvious option was GitHub Pages. It is a free service provided by Github, which nicely integrates with your GitHub repository. If you are using Jekyll as a static site generator, it offers good integration out of the box.

While GitHub pages are good and will do the job, there is one service, which is in my opinion much better. It offers much more while keeping the simplicity. It's called Netlify. This blog post provides a useful comparison of GitHub Pages and Netlify.

Netlify to the rescue

All right, Netlify has many nice features, let's dig a bit deeper.

Simple Setup - Continous Deployment

What I love about Netlify is its simplicity. It can take just a minute to set up continuous deployment from your git repository.

You create a Netlify account by authenticating using you GitHub, GitLab or BitBucket accounts.

You select which git provider (GitHub, GitLab, BitBucket) you want to use.

Then a repository you want to deploy to Netlify.

You select a build command (e.g. gatsby build) and the default branch to build from. Netlify is usually able to prefill this for you by detecting your static code generator from your repo's code.

That's all the setup needed to get up and running. Now Netlify builds your code and publishes it. In some cases, it is all you need. From now on, every time you push new changes to the branch configured above, Netlify builds a new version of your pages using the command provided. Then it deploys this new version and invalidates all the caches. It still keeps the older versions so you can quickly switch to them if anything goes wrong.

If you have a custom domain, you can easily use your own domain name instead of yoursite.netlify.com. After this, you can also add HTTPS with just one click (see below).

Content Delivery Network

One of the weaknesses of my previous WordPress setup was performance. The problem was two-fold. First - having a PHP backend generating and serving my content is way slower than just serving a static HTML page. Static site generators such as GatsbyJs or Jekyll solve this issue. The second problem still remains. If you have just one central server distributing your static pages, the latency increases with the distance of your clients from the central server. Having someone connecting from the other end of the world causes a serious performance hit. What's more, you have a single point of failure. If your central server goes down, your content is not served.

Content Delivery Network (CDN) solves this issue. CND offers many nodes distributed in various geographic locations. When you are requesting a resource from a CDN, you are served from the node which is geographically closest. Moreover, if one of the nodes goes down, the traffic is redirected to the next one. Since your static site is not changing until the next build, it is easy to scale and distribute your whole site over a CDN. Netlify does this out of the box, utilizing multiple intelligent ultra-fast CDNs based on multiple providers. That means a tremendous performance and reliability gain over the original WordPress solution. What's more, Netlify's logic layer on top of the CDN allows instant cache invalidations, rollbacks or custom redirect, rewrite or proxy rules.

One click HTTPS

Having a secure HTTPS connection enabled on a site like a blog, where no sensitive data is transferred, may seem a bit pointless. However, there are several good reasons to use HTTPS anyway.

First of all, sites using HTTPS get a ranking boost in search results on Google. That means your site is displayed above the similar sites not using HTTPS, which can bring much traffic. Moreover, Google Chrome is going to show "Not Secure" on all the pages not using HTTPS soon enough, which can scare off your visitors. When running over HTTP, you also risk public networks injecting content and advertisements into your pages, which can ruin user experience and lead to loss of users.

If you want to use HTTP/2 and enjoy its benefits such as increased performance, you need to migrate to HTTPS. Netlify utilizes Let's Encrypt - a free, automated, and open Certificate Authority.

UPDATE: As of July 2018, All new sites on Netlify are HTTPS by default. You no longer need to explicitly enable it.

Deploy preview per branch

By default, Netlify deploys only changes in one configured branch. However, you can configure it to extend this behavior to multiple branches or even to all of them. In such case, whenever you make changes in such branch, Netlify builds and deploys your pages, but not to your primary domain. Instead, it creates a custom domain just for your branch, where you can check how your site works before going public. You can share it with others to gather feedback on your new version, perform testing and so on. It is handy as you can test your site in a production-like environment before actually publishing it. No more "It works on my machine" excuses.

A/B testing and private betas

Split Testing is one of the cool features, which are so simple to set up on Netlify but hard otherwise. You can configure multiple branches in your repository and set that certain percentage of users is redirected to builds of specific branches.

It can be very useful. You can use analytics to perform A/B testing and determine which version of your site is more effective. You can roll new changes just to a fraction of your users, make sure everything is ok and only then publish the new version globally. You can even do private betas.

Integration with a CMS

WordPress does not only serve your content, but it is also a Content Management System, of course. It may sound like a good idea, but it comes at a price, which is mainly a performance hit. Switching to a static site means much better performance, but you no longer have a nice integrated CMS.

Fortunately, there is plenty of them, which you can use for your JAM Stack static site. Moreover, having decoupled creation and serving of content is very powerful. Also, because you are using git for your site, you can easily utilize its power to achieve scenarios such as editorial workflow, where articles are created as drafts and then need to be approved before publishing. So all is version controlled with all its advantages.

You can use any CMS you want, but if you want to stick with Netlify, you can utilize its Netlify CMS, which is also available as a GatsbyJs Plugin.

And much more

Features above make Netlify such a great and powerful tool, yet easy to use. Netlify, however, offers much more. These more advanced, features are not needed for my simple use case of a personal blog, but they can be valuable in other cases.

Netlify provides support for effortless form submission integration without the need to provide your own backend. Another cool feature is identity management. Netlify provides you with easy integration of processes such as registration, login or account recovery. You can even enable an authenticated user to call third-party services.

You can define redirects, which can be useful when migrating from an existing solution with a different content structure as I did. You are able to even define custom HTTP headers to be returned. When using single page applications, you can turn on pre-rendering of your content for search engine crawlers. Your app then looks just like ordinary HTML site instead of SPA, which can be beneficial for SEO.

The last but not least is a powerful integration of AWS Lambda functions. Everything is managed through Netlify, so you are shielded from coordinating the deployment of your app and server-side functions. No need to worry about service discovery or API gateway configuration. How cool is that?

Conclusion

Migrating to a static site generator such as GatsbyJs is a first (and essential) step. Then, however, you need to figure out how and where to build and deploy your site. Netlify is very easy to use, yet powerful solution offering great performance and user experience. Moreover, for most scenarios (such as my blog) it's completely free.

JShell - New REPL tool in Java 9 for quick prototyping

Fri, 20 Apr 2018 22:12:03 GMT

REPL

REPL stands for Read-Eval-Print-Loop. It may sound a bit cryptic, but it is just a fancy name for an interactive shell for a programming language. Many languages these days already have a REPL available. Even on JVM Groovy, Kotlin, Scala, and Clojure already have it. Java starting with version 9 finally has its very own REPL called JShell (based on Java Enhancement Proposal 222).

Alright, so Java finally has its new shiny REPL. But what is it good for? Well, in a nutshell, it allows you to write independent snippets of Java code to the console (READ), immediately execute them (EVAL), then see the results (PRINT) and continue remembering what you already wrote (LOOP). It is a perfect tool if you want to try a piece of code quickly, draft an algorithm, check how some method behaves for unusual input, create and test a code snippet for your blog post etc. You just quickly try some throwaway code and immediately see the results. The best part is - it does not require most of the Java boilerplate.

While prototyping and quick code verification are important, there is another reason why having a REPL is useful. Especially in a verbose language full of boilerplate such as Java. Imagine you are teaching the Java language to complete beginners. What do you need to write a simple hello world program that just prints to the console? A lot of boilerplate such as a class with the main method. The problem is you need to expose the student to a lot of concepts they don't need to worry about right now such as classes, methods, static, string arrays and so on. Then when you make changes, you need to recompile and rerun the whole thing.

The interactive console does not require class with main and immediately shows you output. You can try various constructs and you see results quickly. You don't need any IDE. You can start programming with minimal setup and minimal knowledge of all the advanced concepts, learning just one construct at the time. This high barrier to entry also resulted in many schools and institutions abandoning Java as their introduction to programming language choice. The educational aspect was actually the primary motivation for the feature as stated in the JEP222:

Immediate feedback is important when learning a programming language and its APIs. The number one reason schools cite for moving away from Java as a teaching language is that other languages have a "REPL" and have far lower bars to an initial "Hello, world!" program.

Running JShell

JShell is bundled with JDK 9+ installation. It resides in your JDK\bin folder. For example on Windows it can be here:

C:\Program Files\Java\jdk-9.0.4\bin\jshell.exe

To run it directly from the console, make sure JDK\bin is added to your PATH. Then simply run jshell command. For now, let's use verbose mode with -v, which will help us better understand what's going on under the hood. Alternatively, you can run the executable directly from the bin directory. The last option is to use your IDE integration (see below - Support In IntelliJ IDEA).

C:\Users\vojtech> jshell -v
| Welcome to JShell -- Version 10.0.1
| For an introduction type: /help intro

jshell>

Now you are in interactive mode and JShell evaluates anything you write. To quit JShell just type /exit.

Expressions

The simplest way to start with JShell is to write a simple expression. It can be a simple mathematical expression:

jshell> 7*(3+12)
$1 ==> 105
|  created scratch variable $1 : int

As you can see the expression is immediately evaluated and the result is printed to the console. No need to declare any variables first. For our convenience, however, a temporary variable called $1 was created, which we can use from now on. Note that type of the variable was inferred to int.

However, expressions are not limited to simple mathematical ones like above. That would not be very useful. You can do pretty much anything you can do in Java.

jshell> Math.sqrt($1)+7
$2 ==> 17.246950765959596

First of all, notice that we were able to use $1 variable from the previous example. The state is preserved between each command. Another noteworthy feature is that, as you can see, semicolons are optional in most cases.

Variables

Even though JShell declares variables for us when the return value is not assigned to any variable, it is usually better to declare your own variables. If only for the sake of descriptive naming. You can declare them as you would local variables.

jshell> int myVariable = 42
myVariable ==> 42
|  created variable myVariable : int

Access modifiers and static/final are ignored for top-level variable and function declarations:

jshell> final int myFinalVariable = 18
|  Warning:
|  Modifier 'final'  not permitted in top-level declarations, ignored
|  final int myFinalVariable = 18;
|  ^---^
myFinalVariable ==> 18
|  created variable myFinalVariable : int

After some time you may get confused what variables you already declared and what are their values. There is a specific command precisely for this - just type /vars.

jshell> /vars
|    int $1 = 105
|    double $2 = 17.246950765959596
|    int myVariable = 42
|    int myFinalVariable = 18

If you are on Java 10 already, you can use var instead of declaring the type explicitly.

Methods

As we saw above, you can declare variables on the root level, not inside any class. You can do the same for methods. Again, you don't need to worry about any modifiers such as public, static or final. You simply start with the return type and the name of the method with possible parameters:

jshell> String sayHello(String name) {
   ...> return "Hello, my name is "+name;
   ...> }
|  created method sayHello(String)

jshell> sayHello("Joe")
$7 ==> "Hello, my name is Joe"
|  created scratch variable $7 : String

In the example above, you can see we declared a method and then called it. Please note that unlike on the top level, inside methods and classes semicolons are not optional.

A common scenario is when a method uses another method or variable, which is not declared yet. It is called the forward reference and JShell allows you to do that. However, you cannot use such methods until all of its dependencies are also declared.

jshell> String myMethod(String name) {
   ...> return otherMethodNotDeclared();
   ...> }
|  created method myMethod(String), however, it cannot be invoked until method otherMethodNotDeclared() is declared

Similar to /vars for variables, you can list all currently declared methods with /methods.

Types

Top-level variables and methods are useful, but often you need to declare and use regular classes, enums or interfaces. You can do it as usual, nothing JShell specific here. Just keep in mind that in this case semicolons are required. You can list all declared types by /types.

jshell> class Person {
   ...>     private String name;
            [more code here]
   ...> }
|  created class Person

Using external code

Defining all your classes as in the example above is a tedious task. What's more, often you want to use already existing classes from JDK or even your own.

For the JDK classes, you can use import as usual. For your convenience, many of the common classes are already imported by default. Not only usual java.lang, but also java.io, java.math, java.util or java.nio.file. You can list all the current imports by /import.

Of course, import is useless if JShell does not have access to the classes needed.

jshell> import com.vojtechruzicka.*;
|  Error:
|  package com.vojtechruzicka does not exist
|  import com.vojtechruzicka.*;
|  ^

Your classes need to be on the classpath. The first option is using CLASSPATH environmental variable. Or you can specify classpath when launching JShell:

jshell --class-path foo-1.0.0.jar

You can provide multiple jars separated by either ; or : depending on your OS. Alternatively, you can define classpath directly from jshell:

jshell> /env -class-path foo-1.0.0.jar

If you are using Java 9 module system, you can specify modules to be imported when starting JShell:

jshell --add-modules some.module

Exceptions

The good news is that JShell handles exceptions well. Whenever an exception occurs, JShell catches it, prints it and your session is not terminated. What's cool is that unlike in regular Java, it does not force you to handle checked exceptions, which removes a lot of try-catch boilerplate.

Saving and loading your work

When you work with JShell in more complicated use cases, it is often handy to be able to save your work and to continue later. Or to simply save your session for future reference. Fortunately JShell supports saving and loading sessions using /save and /open commands:

jshell> /save myfile.jsh

jshell> /open myfile.jsh

Using an External Editor

While you can write and edit everything directly in the JShell console, it is not always the best approach. The editing can be tedious and cumbersome. It is often better to use a dedicated external text editor. To open all the code snippets you entered so far in an external editor, type /edit.

Not very cutting edge, is it? Fortunately, you can configure your own editor to open when executing the /edit command. One way is to sen the environmental variable JSHELLEDITOR. Alternatively, you can set the editor directly from JShell. Use -retain option to persist this setting between sessions.

jshell> /set editor myEditor -retain
|  Editor set to: myEditor

If your editor is not on PATH, you'll need to provide the full path to the executable.

Using JShell programmatically

A very interesting option is to integrate your Java applications with JShell. You can create a JShell instance programmatically and then use it in your app. All the required classes are under jdk.jshell. First, you need to create a JShell instance and then you can use it to evaluate code snippets.

JShell shell = JShell.create();
List<SnippetEvent> events = shell.eval("String hello = \"Hello JShell!\";");

The eval method evaluates the code snippet and returns you a list of events, which occurred during the evaluation. From these, you can tell whether some exceptions occurred and whether the snippet is valid. To access details about the snippet itself, you need to call snippetEvent.snippet().

JShell shell = JShell.create();
List<SnippetEvent> events = shell.eval("String hello = \"Hello JShell!\";");
SnippetEvent event = events.get(0);
Snippet snippet = event.snippet();
Snippet.Kind kind = snippet.kind();
String source = snippet.source();

To obtain the full source code of the snippet, you can use snippet.source(). To determine the type of the snippet (variable declaration, importm, method declaration, ...) you can use snippet.kind().

Alternatives

The first interesting alternative to plain JShell is called Try Artifact. Instead of obtaining jar dependencies manually and adding them to the classpath, it allows you to download and use Maven artifacts directly from the console.

jshell> /resolve org.apache.commons:commons-lang3:jar:3.4
|  Path /home/kawasima/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar added to classpath

Another useful alternative is using a REPL directly in the browser. You can try either repl.it or ~~javarepl.com~~. UPDATE: Looks like javarepl is no longer available.

Support In IntelliJ IDEA

The good news is that if you use IntelliJ IDEA, you don't need to worry about having JShell on the classpath, as IDEA offers excellent integration with JShell out of the box directly in the IDE. You get all the useful features such as code completion, syntax highlighting, error detection and more.

To access JShell from IDEA, go to Tools → JShell Console....

What's also useful is that you can prepare all the code in advance and then just run it on demand rather than in the interactive console mode, which is often much more convenient.

Unlike when running directly from the console, IDEA automatically adds your current project to the classpath so you can work with your custom classes out of the box with no setup needed.

Conclusion

Finally, even Java has its own REPL called JShell. It is a useful tool for quick prototyping, teaching or demonstration purposes. It is easy to use as it removes the need of much of the boilerplate Java normally requires. And if you use an IDE, which integrates JShell, the whole process gets even easier.

Getting rid of web.xml in Spring MVC App

Mon, 02 Apr 2018 22:12:03 GMT

From Servlet 3.0 on, web.xml is optional. How to get rid of it in your Spring MVC app?

web.xml and Servlet 3.0

Web.xml, also known as deployment descriptor, is traditionally used as a configuration file for Java web applications. It defines servlets, their mappings, servlet filters, lifecycle listeners and more. Originally it was the only way to provide such configuration. Over the time, once popular XML configuration lost its appeal and popularity in favor of Java-based annotation configuration. The same trend can also be observed in Spring Framework.

With Servlet version 3.0, the deployment descriptor is no longer mandatory. The configuration originally provided by the file can now be provided by annotations such as @WebServlet, @WebFilter or @WebListener directly on the class level. For example, you can annotate your servlet like this to provide mapping and init params:

@WebServlet(value="/my-servlet",
            initParams = {
                @WebInitParam(name="first", value="some-value")
                @WebInitParam(name="second", value="some-other-value")
            })
public class MyServlet extends HttpServlet {
  ...
}

For more info about replacing web.xml with annotations see Servlet 3.0 Tutorial: @WebListener, @WebServlet, @WebFilter and @WebInitParam.

web.xml and Spring MVC

In Spring MVC, web.xml used to be the place, where you needed to declare and configure Dispatcher Servlet, which is a Front Controller, receiving all the requests and dispatching to all the other components such as Controllers. Fortunately, Spring offers a convenient, XML-free way of declaring Dispatcher Servlet.

Interface WebApplicationInitializer

In version 3.1, Spring introduced WebApplicationInitializer. It is an interface, which you can implement. If you do so, Spring will detect your class and execute its method onStartup(ServletContext container), inside which you can define Configuration of your Dispatcher Servlet and do all the other config such as registering and mapping of other servlets, filters or lifecycle listeners. Most importantly, inside the method, you'll need to create your application context. This usually means having root application context and web application context. The specific class you'll need to use for your app context will depend whether you still use XML configuration or Java configuration.

XML Config

To create application context based on XML configuration files, you'll need to use XmlWebApplicationContext.

public class XmlWebAppInitializer implements WebApplicationInitializer {

    @Override
    public void onStartup(ServletContext container) throws ServletException {
        XmlWebApplicationContext rootContext = new XmlWebApplicationContext();
        rootContext.setConfigLocation("/WEB-INF/config/root-context.xml");

        container.addListener(new ContextLoaderListener(rootContext));

        XmlWebApplicationContext dispatcherContext = new XmlWebApplicationContext();
        dispatcherContext.setConfigLocation("/WEB-INF/config/servlet-context.xml");

        ServletRegistration.Dynamic dispatcher = container.addServlet("dispatcher", new DispatcherServlet(dispatcherContext));
        dispatcher.setLoadOnStartup(1);
        dispatcher.addMapping("/");
    }
}

Java Config

If you are already using Java Configuration files in favor of XML or you want to migrate, you'll need application context which will load the configuration from annotation-based java classes. You'll need to use AnnotationConfigWebApplicationContext instead of XmlWebApplicationContext.

public class AnnotationWebAppInitializer implements WebApplicationInitializer {

    @Override
    public void onStartup(ServletContext container) throws ServletException {
        AnnotationConfigWebApplicationContext rootContext = new AnnotationConfigWebApplicationContext();
        rootContext.register(RootConfig.class);

        container.addListener(new ContextLoaderListener(rootContext));

        AnnotationConfigWebApplicationContext dispatcherContext = new AnnotationConfigWebApplicationContext();
        dispatcherContext.register(WebConfig.class);

        ServletRegistration.Dynamic dispatcher = container.addServlet("dispatcher", new DispatcherServlet(dispatcherContext));
        dispatcher.setLoadOnStartup(1);
        dispatcher.addMapping("/");
    }
}

More servlets, filters and listeners

So far so good, our Dispatcher Servlet is configured. But web.xml is not used only for the dispatcher, but for all the servlets, their filters, listeners and more. How to configure these? Just call a corresponding method on the ServletContext object, which is an input parameter of the method:

@Override
public void onStartup(ServletContext container) throws ServletException {

    // Add a Listener
    container.addListener(MyListener.class);
    
    // Add a Filter
    FilterRegistration.Dynamic filter = container.addFilter("FilterName", MyFilter.class);
    filter.addMappingForUrlPatterns(null, true, "/*");
    
    //Add a Servlet
    ServletRegistration.Dynamic servlet = container.addServlet("ServletName", MyServlet.class);
    servlet.setLoadOnStartup(2);
    servlet.addMapping("/foo", "/bar");
    
    ...
}

WebApplicationInitializer implementations

Like in the examples above, you can implement WebApplicationInitializer interface all by yourself and you have complete freedom. However, Spring already provides several implementations you can build on. They are usually abstract classes implementing the interface. They take opinionated approach providing most of the configuration for you letting you specify the details. They basically act as templates forcing you to implement necessary details and allowing you to override the rest if necessary.

For example, there is SpringBootServletInitializer for Spring Boot applications. For regular Spring web apps, there is AbstractAnnotationConfigDispatcherServletInitializer. It will create and register Dispatcher Servlet and root and web application context based on Java classes with annotations. You provide configuration classes and URLs to which Dispatcher Servlet should be mapped to:

public class MyWebAppInitializer extends AbstractAnnotationConfigDispatcherServletInitializer {

  @Override
  protected Class<?>[] getRootConfigClasses() {
    //Provide configuration classes for Root App Context
  }

  @Override
  protected Class<?>[] getServletConfigClasses() {
    //Provide configuration classes for Web App Context
  }

  @Override
  protected String[] getServletMappings() {
    //Provide URLs to which Dispatcher Servlet Should be mapped to
  }
}

Of course, if you need to customize how Dispatcher Servlet is created or any other details, you can override methods from the parent class.

Updating your Maven WAR plugin

After removing web.xml file, your Maven build may break complaining that the file is missing. This happens when you're using an older version of Maven WAR Plugin. You can set configuration flag failOnMissingWebXml to false to fix the error.

<build>
    <plugins>
        <plugin>
            <artifactId>maven-war-plugin</artifactId>
            <version>2.4</version>
            <configuration>
                <failOnMissingWebXml>false</failOnMissingWebXml>    
            </configuration>
        </plugin>
    </plugins>
</build>

Or better yet, if you upgrade your plugin version to 3+, the default value of failOnMissingWebXml is now set to false and does not need to be explicitly specified anymore.

Conclusion

If you have a legacy Spring MVC application, which still uses web.xml, you can remove the file if you're on Servlet 3.0+. One option is to implement WebApplicationInitializer directly. This gives you the most flexibility, but all the work is on you. Another option is to use any of the abstract classes provided by Spring, which implement the interface. In most cases, it is the preferred solution as it leaves you with less manual configuration and it uses reasonable opinionated defaults.

Either way, Java configuration is in many cases preferable as it gives you more flexibility and unlike in XML, you can use conditional logic based on current circumstances.

Migration to GatsbyJS and JAM stack from WordPress

Sun, 25 Mar 2018 22:12:03 GMT

How and Why I migrated from WordPress to static JAM Stack site built with Gatsby JS.

Before

When I started my blog back in 2016, I wanted to focus as much as possible on creating content rather than creating the blog infrastructure. Of course, there was a siren's call to build everything from scratch or spend weeks researching all the possible platforms. It would be, no doubt, a lot of fun. But I resisted as I wanted to start writing as soon as possible. I had wanted to start my blog for a few years already so I wanted to reduce the barrier of entry as much as possible. I knew the most critical part is to build a habit of writing regularly and then I can polish my blog tech stack later.

So WordPress was the obvious choice for me because it is widespread, with a lot of resources and plugins available. And it worked well. More or less. There were issues though. Most notable one was performance. WordPress is sluggish. Especially once you add some plugins. The results of a Lighthouse audit were disappointing:

Especially the performance. With such long page load times, many users just leave before the page fully loads. Especially when on mobile and low-quality connections. Another issue is security. With a significant part of web powered by WordPress, it is a target of automated attacks all the time. That means constant fear of losing your precious blog. And it's internal database storing all the content was pretty much a black box for me. And of course, there's cost involved. As I didn't want to manage my installation, keep it updated and in good shape, I paid for hosted solution. Last but not least, Wordpress is built with PHP, which I am not familiar with and it is something I am not eager to learn. That means that any customizations or tweaks are out of the question for me.

Long story short, I decided to get rid of my WordPress and replace it with a much more lightweight solution. There was a brief moment where I considered using Medium instead, but I decided I value control of my own content more. After listening to a very inspiring lightning talk by Ladislav Prskavec about JAM Stack, I decided its time to give it a try.

JAM Stack

JAM is an acronym for Javascript API Markup. What does it mean though? It is not a framework or a specific technology. It is more of an architectural approach. A modern way web content is built, managed and distributed.

Back in the day, the web was just static HTML pages, maybe with some CSS and Javascript on the top. Perfect for static content. However, usually, you need your data to be more dynamic. Instead of static HTML sitting on the server served directly, HTML pages were built on-the-fly when a client requested them. The data dynamically provided from a database. While this is very flexible, it comes at a cost. It's slow. Way slower than just a static site. The application server executes code on demand, needs to contact a database, usually on a different machine, build the page and send it back. The scalability is hard and expensive. When the traffic rises, its more complicated to scale the solution. We're talking load balancing, session replication, distributed caching not to mention difficult RDS scalability. And it's vulnerable. With so many moving parts, the attack surface is extensive. Ranging from Cross-Site-Scripting, SQL Injection to exploiting security vulnerabilities in your libraries.

When you have just static HTML, CSS and Javascript files, it's much harder to compromise your site unless you gain complete access to your server. Since the content does not change, it is easy to cache. Instead of having just one central server or just a few of them in an expensive and complicated solution, you can use Content Delivery Network to distribute your site. This way you'll have your availability covered and CDN redistributing your files across the globe. Users from New York will be served as fast as users from Tokyo. And if your traffic skyrockets, it is still not a problem. Scalability is easy and cheap if you don't have to manage synchronization of state.

One of the key parts of JAM Stack Architecture is switching to static. That means, using a Static Site Generator to build your site into plain old HTML, CSS, and JS, which does not change and can be served over CDN. If you need to make a change, you just rebuild the whole site. It is fast since most of the site remains the same. Moreover, the deploy takes very little time as it means only copying a bunch of files.

The source content files are usually not HTML, but some templating or markup language (that's M in JAM). For example, you can write all the content in Markdown or AsciiDoc and convert it to HTML at the build time. That's simpler than writing directly in HTML and more secure because of XSS. And even a non-technical person can learn it quickly.

One important part to realize is that static does not mean limited interactivity and static data. The key part is that HTML and JS files served by the server are static. They're not changing. However, you still have Javascript on the client. You can do everything you do in JS. That means you can fetch any data you need through an API call (usually REST or GraphQL) once your static page is loaded. The key difference is that the HTML with dynamic data is not constructed on the server, but rather on the client. Of course, in many cases, your site is really static and no API calls are necessary.

Another key part is that everything is in Git, no database, no App server with configuration, just your code which is built by a static site generator. And everything builds and deploys quickly. That allows you to automate the build and deployment workflow easily. When you make changes in the master branch, it automatically builds the site with your Static Site Generator and just replaces the old static files. That's it. That means easy continuous deployment. And there are services for this.

GatsbyJS

Alright, JAM Stack is a way to go, no doubt about that. Which of the static site generator to use though? A good place to start is a site called StaticGen, which contains a nice list of many Static Site Generators. You can filter by language and templates used. And sort by popularity. That's a good way to determine what generators are currently widely used, which is important because it determines community size, resources, and plugins available. Currently, there are 214 generators and the number is constantly growing. So which one should I choose?

There are many factors to consider. Maturity is important as it means a big community, lot of resources, plugins and stable bug-free product. On the other hand, it is often a trade-off between maturity and new features and technologies. Then, of course, there is the language used. If you need to customize the generator and tinker with it, you'll want to use one written in a language you're familiar with. After briefly checking top 5 generators, I declined Jekyll and Hugo as Go and Ruby were not languages I am proficient with. From the remaining generators, Gatsby immediately caught my attention because of its features and modern technologies.

Modern technologies

Gatsby is built with a lot of interesting technologies. If you are a front-end developer, you'll probably be familiar with most of these, so adoption should be quick. If not, it gives you chance to learn a lot of interesting stuff. Gatsby is powered by React. You'll be able to nicely structure your project to components and use all the existing React components there not specifically developed for Gatsby. For styling, you can attach plain old CSS stylesheet, but Gatsby offers many more choices like SASS, Glamor, Styled Components, Stylus, LESS or Styled JSX. Gatsby is built on top of Webpack. And it can generate offline-ready Progressive Web App and uses GraphQL for querying data.

Data sources and GraphQL

One of the disadvantages of WordPress is that is tightly couples creation and management of content and serving of the content. While for many users WP may be a nice way to create and manage their content, it is far from efficient for serving it compared to static sites. In contrast, this is one of the areas where Gatsby really shines.

From your react components and templates, you access data (such as blog posts, site title and other metadata) using GraphQL. Unlike other Static Site Generators, the data is not limited to Markdown or some templating language. Gatsby supports many sources. It can handle files in formats such as Markdown, YAML, CSV or XML. What's awesome is that it can connect to sources like Medium or WordPress and load data from there! And because of Gatsby's extensibility and plugin system, it is easy to add more sources. What's great is that from the perspective of your templates it is just a GraphQL query. That means it is easy to switch data sources while your components and templates remain intact. For example, when migrating from WordPress, you can keep your articles in WP, keep creating and updating them there and just use Gatsby to transform your WP site to static one. Then, later you can convert to markdown if you feel like it.

What's really powerful is using GraphQL as the query language. Because when querying structured data, you define what fields are you interested in, you can define order or limit the number of the results. But everything is defined on the side of the client executing the query. You can change your queries and data obtained without modifying the data source. It also means you can query the same source in different ways depending on context.

Performance

One of the Gatsby's greatest advantages is performance. I mean - performance on top of what you get by switching to a static site generator. Since Gatsby uses Webpack, it allows code splitting by route. That means when accessing a page on your site, the client will download only code and resources necessary for the current sub-page and not the whole bundle. When navigating to another route, it will just fetch what's new. Gatsby also sticks to the Progressive Web App standard. That means your site can behave as a native-like app on mobile and work with a poor connection or even offline. What's more, Gatsby utilizes PWA's PRPL Pattern. That means first push critical resources over HTTP/2 to the client, which is faster than waiting for the browser to download HTML first, parse it and then request additional stylesheets, scripts and resources linked in the HTML. Gatsby first renders static HTML version of the initial route to minimize time-to-interactive. Then in the background, it loads the bundle and initializes Single-Page App. It starts to pre-fetch resources for the routes linked to the initial page so additional navigation within your site is blazing fast and without reload.

Gatsby also works great with images, which are usually a big hit to site's performance. Based on the screen size and resolution of the client device, it will serve appropriate image size. It supports WebP and can convert your jpegs to progressive jpegs. While the images are loading, it will reserve the space required by the image, so the content of the page is not 'hopping' after image loads. It also replaces the image with a blurred version or monochromatic SVG until it loads - similar to what Medium uses.

Pitfalls

I think Gatsby is great. I had a lot of fun using it so far and I've learned much in the process. It offers you freedom and flexibility. However, it is not as mature as some of the competitors. This is improving quickly as it is under rapid development. Still, I did encounter some bug complicating my development. On the other hand, the community was really helpful and supportive. Some of the features, which I would consider standard were not yet available or sufficient such as tags and categories support, pagination or Open Graph metadata. I would also like to see AsciiDoc support instead of just plain Markdown. One of the biggest problems was probably the lack of visual themes. For a long time, there was not any official themes directory. In Feb 2018 Gatsby Manor was launched and now it contains just SIX themes, which are all only ports of HTML5 UP themes. However, with the current development rate, I am sure more and more plugins will be available out of the box shortly.

Results

Overall I am really happy with Gatsby and the migration from WordPress. I no longer fear my WordPress will get compromised. Everything is now on GitHub and under version control. It's much more transparent. No longer need of database backups and rollbacks are now easy in case of something goes wrong. Not to mention the possibility of developing new features safely in branches and deploying and testing them separately from the 'production' environment. And no more fees for hosted WordPress. And writing in markdown is also more convenient for me as I don't focus on HTML or visuals when writing but rather on the content.

My blog is now deployed on Netlify CDN, which means the user will be served fast no matter what their geographic location is. The performance gain of switching to static is huge. That means not only fewer visitors leaving the blog because of the slow loading times, but also better page ranking on Google as it favors faster sites. It also favors sites served over HTTPS, which my site now supports thanks to Netlify's one click setup and Let's Encrypt. Now let's go back to the Lighthouse audit score and compare new Gatsby site to my good old WordPress.

Before

After

Quite a difference, right? Long story short - Gatsby is great. You should give it a try, especially if you are a front-end developer. And keep watching its development as I am sure we can look forward to even more awesomeness in the future. If Gatsby is not your cup of the, you should still consider switching to a static site generator as the advantages are too good to ignore.

To learn more about Gatsby check out this curated list of Gatsby-related resources.

Deployment

This article covers GatsbyJS part of the migration. But that's not all you need to do, you also need to set up a build and deployment process for yor site.

I decided to use Netlify to set up a continuous deployment of my site, you can learn more here: Migration to JAM stack and Netlify from WordPress.

Documenting Spring Boot REST API with Swagger and SpringFox

Fri, 16 Feb 2018 22:12:03 GMT

How to document your Spring Boot REST APIs using Swagger with SpringFox?

Swagger and SpringFox

Documenting your REST API is very important. It is a public interface, which other modules, applications or developers can use. Even if you're not publicly exposing it, it is still important. Backend and frontend code is usually worked on by different developers. The one who is creating the API is usually not the one who is consuming it. It is, therefore, crucial to have properly documented interface to avoid confusion and keep it always up to date.

One of the most popular API documentation specifications is OpenApi, formerly known as Swagger. It allows you to describe your API's properties using either JSON or YAML metadata. It also provides a web UI, which is able to turn the metadata into a nice HTML documentation. What's more, from that UI you can not only browse information about your API endpoint, but you can use the UI as a REST client - you can call any of your endpoints, specify the data to be sent and inspect the response. It's quite handy.

It is however not realistic to write such documentation by hand and keep it updated whenever your code changes. This is where SpringFox comes into play. It is a Swagger integration for Spring Framework. It can automatically inspect your classes, detect Controllers, their methods, model classes they use and URLs to which they are mapped. Without any handwritten documentation, it can generate a lot of information about your API just by inspecting classes in your application. How cool is that? Most importantly, whenever you make changes they'll be reflected in the documentation.

Starting project

To start, you'll need a Spring Boot application with some Rest Controllers, I've prepared a simple one here.

For this article, I used SpringFox 2.9.2 and Spring Boot 1.5.10.RELEASE. It uses version 2 of the Swagger specification. Version 3 is already out, but it is not yet (as of 2/2018) supported by SpringFox. The support should be available in the next version.

The source code of final project built with all the features described in this blog post is available on GitHub.

Adding Dependencies

To work with SpringFox in your project, you need to add it as a dependency first. If you are using Maven, you can use the following (you can check whether a newer version is available).

<dependency>
    <groupId>io.springfox</groupId>
    <artifactId>springfox-swagger2</artifactId>
    <version>2.9.2</version>
</dependency>

Or if you are using Gradle:

compile "io.springfox:springfox-swagger2:2.9.2"

Basic configuration

After adding the dependency, you'll need to provide some basic Spring configuration. While you can technically use one of your existing configuration files, it is better to have a separate file for it. The first thing you'll need to provide is a @EnableSwagger2 annotation. Then you need to provide a Docket bean, which is the main bean used to configure SpringFox.

@Configuration
@EnableSwagger2
public class SpringFoxConfig {
    @Bean
    public Docket apiDocket() {
        return new Docket(DocumentationType.SWAGGER_2)
                .select()
                .apis(RequestHandlerSelectors.any())
                .paths(PathSelectors.any())
                .build();
    }
}

Of course, you can provide many more configuration settings as we'll see later, but this is a minimalistic configuration, which does the following:

@EnableSwagger2 enables SpringFox support for Swagger 2.
DocumentationType.SWAGGER_2 tells the Docket bean that we are using version 2 of Swagger specification.
select() creates a builder, which is used to define which controllers and which of their methods should be included in the generated documentation.
apis() defines the classes (controller and model classes) to be included. Here we are including all of them, but you can limit them by a base package, class annotations and more.
paths() allow you to define which controller's methods should be included based on their path mappings. We are now including all of them but you can limit it using regex and more.

Adding UI

If you deploy your application now, swagger metadata describing your API is already being generated! You can check it out:

http://localhost:8080/v2/api-docs

Turns out it is just a big JSON, not very human readable. But you can already verify it works. Just go to the Swagger Online Editor and paste the JSON there. Paste your generated JSON to the left panel and voila! You can now see your generated documentation as HTML page. Nice, isn't it? It would be even nicer to have such documentation directly as a part of your application. Fortunately, it is quite easy to achieve this. The GUI displaying HTML documentation based on JSON input is called swagger-ui. To enable it is a Spring Boot app, you just need to add this dependency:

//MAVEN
<dependency>
    <groupId>io.springfox</groupId>
    <artifactId>springfox-swagger-ui</artifactId>
    <version>2.9.2</version>
</dependency>

//GRADLE 
compile "io.springfox:springfox-swagger-ui:2.9.2"

The documentation will be automatically available here:

http://localhost:8080/swagger-ui.html

Adding ApiInfo

By default, the header part of our documentation does look pretty generic:

It's time to do something about it. We can change all the information there just by a simple configuration change. In the SpringFoxConfiguration file, we need to add ApiInfo object, which provides general information about the API such as title, version, contact or licensing information.

@Bean
public Docket apiDocket() {
    return new Docket(DocumentationType.SWAGGER_2)
            .select()
            .apis(RequestHandlerSelectors.any())
            .paths(PathSelectors.any())
            .build()
            .apiInfo(getApiInfo());
}

private ApiInfo getApiInfo() {
    return new ApiInfo(
            "TITLE",
            "DESCIPRION",
            "VERSION",
            "TERMS OF SERVICE URL",
            new Contact("NAME","URL","EMAIL"),
            "LICENSE",
            "LICENSE URL",
            Collections.emptyList()
    );
}

Now our documentation header should look much better:

Narrowing down processed APIs

So far so good. But when you take a closer look at the generated documentation, you'll see that in addition to our Model and Controller classes which we use, there are also some spring specific classes such as BasicErrorController in the Controllers' sections and also View and ModelAndView under Models section.

Sometimes it is useful to narrow down classes which will SpringFox detect as sources for documentation generation. Both Controller and Model classes. You can easily configure this in the Docket configuration. Remember like we used .apis(RequestHandlerSelectors.any() to include all the classes? Let's narrow it down just to our base package:

@Bean
public Docket apiDocket() {
    return new Docket(DocumentationType.SWAGGER_2)
            .select()
            .apis(RequestHandlerSelectors.basePackage("com.vojtechruzicka"))
            .paths(PathSelectors.any())
            .build()
            .apiInfo(getApiInfo());
}

This is useful when you want to specify which classes should be included. Sometimes you also need to include only specific URL paths. Maybe you are using multiple versions of your API for backward compatibility but don't want to include the historical ones. Maybe some part of the API is internal and should not be part of the public documentation. Either way, such inclusion based on URL matching can also be configured in the Docket. Remember .paths(PathSelectors.any())? Instead of any, which matches all the paths, you can limit it just to some regex or Ant-style path patterns.

@Bean
public Docket apiDocket() {
    return new Docket(DocumentationType.SWAGGER_2)
            .select()
            .apis(RequestHandlerSelectors.basePackage("com.vojtechruzicka"))
            .paths(PathSelectors.ant("/v2/**"))
            .build()
            .apiInfo(getApiInfo());
}

In case built-in options are not enough for you, you can always provide your own predicate for both apis() and paths(). An alternative way of ignoring certain classes or methods is to annotate them with @ApiIgnore.

Using JSR-303 annotations

JSR 303: Bean Validation allows you to annotate fields of your Java classes to declare constraints and validation rules. You can annotate individual fields with rules such as -- cannot be null, minimal value, maximal value, regular expression match and so on.

public class Person {
    @NotNull
    private int id;

    @NotBlank
    @Size(min = 1, max = 20)
    private String firstName;

    @NotBlank
    @Pattern(regexp ="[SOME REGULAR EXPRESSION]")
    private String lastName;

    @Min(0)
    @Max(100)
    private int age;

    //... Constructor, getters, setters, ...
}

This is a common practice which is already widely used. The good news is that SpringFox can generate Swagger documentation based on such annotations, so you can utilize what you already have in your project without writing all the constraints manually! It is very useful as consumers of your API know what are restrictions on the values they should provide to your API and what values to expect. Without the inclusion of such annotations, the generated documentation for our person model looks rather plain, nothing except for field names and their data type.

With data from JSR-303 annotations, it will look much better:

Unfortunately, JSR-303 based documentation does not work out of the box, you need an additional dependency:

//MAVEN
<dependency>
   <groupId>io.springfox</groupId>
   <artifactId>springfox-bean-validators</artifactId>
   <version>2.9.2</version>
</dependency>
 
//GRADLE
compile "io.springfox:springfox-bean-validators:2.9.2"

And you need to import BeanValidatorPluginsConfiguration configuration file on top of your swagger configuration class:

@Configuration
@EnableSwagger2
@Import(BeanValidatorPluginsConfiguration.class)
public class SpringFoxConfig {
  ...
}

Adding Swagger Core annotations to your model classes

The advantage of using JSR-303 is that if you already use them you get extra documentation information with zero effort and without changing any code. The problem is that currently, SpringFox does not display validation messages specified in the annotations. Also, you may have some more complicated constraints you need to document. In such cases, you can use Swagger Core annotations, which allow you to specify additional information such as description. Person Class annotated with these annotations can look something like this.

@ApiModel(description = "Class representing a person tracked by the application.")
public class Person {
    @ApiModelProperty(notes = "Unique identifier of the person. No two persons can have the same id.", example = "1", required = true, position = 0)
    private int id;
    @ApiModelProperty(notes = "First name of the person.", example = "John", required = true, position = 1)
    private String firstName;
    @ApiModelProperty(notes = "Last name of the person.", example = "Doe", required = true, position = 2)
    private String lastName;
    @ApiModelProperty(notes = "Age of the person. Non-negative integer", example = "42", position = 3)
    private int age;

    // … Constructor, getters, setters, ...
}

On the class level, you use @ApiModel annotation and on field level @ApiModelProperty. You can, of course, mix and match with JSR-303 annotations. @ApiModelProperty's example is useful for providing example values, which is good not only for the guidance of users but also it is used to prefill a request payload when using Swagger UI as a REST client to test your services. Position attribute is handy to specify the order in which attributes will be displayed in the documentation. It is useful to provide important or required attributes first or group attributes which belong together. Otherwise, the attributes will be listed alphabetically.

Adding Swagger Core annotations to your controller classes

Same as you could annotate your model classes with Swagger core annotations to provide additional metadata, you can annotate your controllers and their methods and method parameters.

@Api describes the whole controller
@ApiOperation is used for description on a methods level
@ApiParam is used for method parameters

@RestController
@RequestMapping("/v2/persons/")
@Api(description = "Set of endpoints for Creating, Retrieving, Updating and Deleting of Persons.")
public class PersonController {

    private PersonService personService;

    @RequestMapping(method = RequestMethod.GET, produces = "application/json")
    @ApiOperation("Returns list of all Persons in the system.")
    public List getAllPersons() {
        return personService.getAllPersons();
    }

    @RequestMapping(method = RequestMethod.GET, path = "/{id}", produces = "application/json")
    @ApiOperation("Returns a specific person by their identifier. 404 if does not exist.")
    public Person getPersonById(@ApiParam("Id of the person to be obtained. Cannot be empty.")
                                    @PathVariable int id) {
        return personService.getPersonById(id);
    }

    @RequestMapping(method = RequestMethod.DELETE, path = "/{id}")
    @ApiOperation("Deletes a person from the system. 404 if the person's identifier is not found.")
    public void deletePerson(@ApiParam("Id of the person to be deleted. Cannot be empty.")
                                 @PathVariable int id) {
        personService.deletePerson(id);
    }

    @RequestMapping(method = RequestMethod.POST, produces = "application/json")
    @ApiOperation("Creates a new person.")
    public Person createPerson(@ApiParam("Person information for a new person to be created.")
                                   @RequestBody Person person) {
        return personService.createPerson(person);
    }

    @Autowired
    public void setPersonService(PersonService personService) {
        this.personService = personService;
    }
}

Now your documentation should contain also the descriptions provided:

Note that our controller and domain classes are now plagued with Swagger specific annotations. The readability suffers a lot as the important information gets lost in a lot of fluff. What's worse - documentation written this way does not get updated when you change the code, you'll need to remember to change the messages manually. This increases the risk of your docs being out of sync and thus not trustworthy. It is good to include just the essential information which is not obvious and which is not already covered well by auto-generated information. Having descriptive names of parameters along with JSR-303 annotations can usually document most of the required information.

Loading Description from properties files

Providing descriptions directly in the annotations is not very elegant. It can take a lot of space, polluting your code. You cannot really support multiple languages. When you want to fix a typo or make some changes to the documentation, you need to rebuild and redeploy your whole application. You cannot have different values based on environment. Not very flexible. Fortunately, Spring provides a concept of Property placeholders. In short, it allows you to provide a placeholder ${placeholder} instead of a hardcoded value. Then you define the value of the placeholder in a .properties file. Spring loads the data from the properties and injects it instead of the placeholder. What's cool is that you can provide multiple property files for each language one. You can provide different property files in different environments. They can just be on the classpath so you don't have to rebuild and redeploy the whole app, just change the property file.

SpringFox supports this mechanism in some of the annotations. It is a nice way to decouple your documentation from your code and have a bit more flexibility. Unfortunately, currently, there is support only for some of the annotations. So for example in the model, they support it on method level (@ApiModelProperty), but not on class level (@ApiModel).

To make this work you need to:

Create a property file, e.g. swagger.properties
Enter your desired messages as key-value pairs where key will be used as placeholder - e.g. person.id=Unique identifier of the person
Instead of annotation text insert a placeholder - e.g. ${person.id}
Register the property file in your configuration on class level - eg. @PropertySource("classpath:swagger.properties")

Alternatives

SpringFox and Swagger are a solid choice. However, you may want to try an alternative before choosing it. There is a very interesting project, which is actually a part of the Spring Framework. It's called Spring Rest Docs. One of its advantages is that it is tightly integrated with your tests. That means it can make sure your documentation is always up to date. Otherwise, your tests no longer pass. Add a request param without documenting it and your tests are no longer green. Remove a param without updating your docs and your tests fail. You can learn more in the following article:

Spring REST Docs - Test driven documentation of your REST API

31 May, 2018

Test driven REST API documentation as an alternative to traditional Swagger docs.

Conclusion

SpringFox is a useful tool, which can automatically generate Swagger documentation based on your Spring controller and model classes. It can also recognize JSR-303 annotations, so you'll have also documented all the constraints on your model classes. It can also utilize core swagger classes such as @ApiModelProperty. Be careful though as this plagues your code with a lot of swagger specific annotations. It is always better to use them only when SpringFox cannot infer the information itself. Use them only when you need to add some description where the class, attribute, and methods names are not self-explanatory. Then again, it may be a red flag that your API is cryptic or too complicated. If you leave the majority of your documentation automatically generated by SpringFox, you can be sure that it is always up to date. Otherwise, you need to be really careful to update the Core Swagger annotations when making changes in your code. If your docs and code are not matching, users will lose trust in your API documentation and such documentation is next to useless.

Javascript: Uncovering mysteries of ‘this’ keyword

Tue, 13 Feb 2018 22:12:03 GMT

'This' is a very straightforward concept in other languages. Not so in javascript. It can point to pretty much anything depending on the context.

What the hell is this?

If you are coming from another language, which is Object Oriented, such as Java, you are no doubt familiar with the concept of this. This keyword is used inside of a class and refers to the current instance. No more, no less.

In Javascript, this is yet another concept, which behaves in an unexpected way. Like other concepts such as new operator or classes, it tricks you into making some false assumptions based on concepts which you know from other languages. Because the syntax and naming are the same or very similar, yet the concept is different. What's more, its behavior is not the same depending on whether you are in the strict mode or not.

Every time a function is invoked, this is assigned a reference to an object based on how the function was called. Is it a regular function? A method called on an object? Fat arrow function? Method of an ES6 class? That's the key to understanding the topic. What's confusing about this is that the very same function can have a different value of this based on circumstances. Turns out that this does not depend on the function itself but rather on how the function is called. The same function will have different this when called as a method of an object and when provided as a callback function. You need to be careful because the functions can be passed around and in such case, this may be different from what you expected.

Function invocation

When calling a function this is referring to the global object. The global object depends on how you run your code -- if you are executing in a browser, it is the window object. In node.js it is an object called global.

function logThis() {
  console.log(this);
}

logThis(); // global object - e.g. window

Note that it applies to function calls, where the function is not called as a property of an object. eg. object.function(). In such cases, it is actually method invocation, which follows different rules (see below).

Alright, so far so good. This in functions calls points to the global object. It is not so simple though. This behavior only applies when not in strict mode. When in strict mode, this in functions is undefined.

function logThisInStrictMode() {
  "use strict";
  console.log(this);
}

logThisInStrictMode(); // undefined

Method invocation

A method is basically a function, which is a property of an object. In methods, the situation is simple -- this refers to the object the method is owned by.

var john = {
  name: 'John',
  greet: function() {
      // this points to the enclosing object
      console.log('Hi! My name is ' + this.name);
  }
};

john.greet(); //methods are called through their object (john here)

Be careful though. That does not apply to nested functions. If you declare a function inside an object's method, its this does not point to the original method's object. It is a regular function, therefore, it's this points to the global object (see above). Unless, of course, you are in strict mode. In such case it is undefined.

var john = {
    name: 'John',
    greet: function(people) {
        people.forEach(function (person) {
            // this points to the global object
            console.log('Hi ' + person + '. My name is ' + this.name);
        });
    }
}

john.greet(['Jane', 'James', 'Jill']); // Hi Jane. My name is undefined ...

Alright, not good. But what can we do to fix it? We need this to point to the john object! There are some ways solve this issue. One common approach is storing this reference in a variable (by convention usually called that). Since functions have access to the variables declared in their outer function, we can then easily access that variable from the inner function. And it will still keep the original this which we need.

var john = {
    name: 'John',
    greet: function(people) {
        var that = this;
        people.forEach(function (person) {
            // this points to the global object
            console.log('Hi ' + person + '. My name is ' + that.name);
        });
    }
};

john.greet(['Jane', 'James', 'Jill']);

This works, but there are more solutions, which we'll cover later. For now, keep in mind that this points to the owner object ONLY when you call the method through the object:

john.greet(); //finally works

If you take the very same method and call it in a different way, this will be different. Let's say we store the method to a variable and then invoke it. Suddenly, when it is not called on an object, it is just an ordinary function and, as we know, the function's this points to the global object.

var greetingFunction = john.greet;
greetingFunction(); // Hi! My name is undefined

In constructor functions

When you are using constructor functions with a new keyword, this behaves a bit differently than usual. In short, what new operator does is that:

It creates a new blank object.
It makes this to point to this newly created object inside the constructor function
It sets the prototype of the newly created object to the constructor function's prototype.
It makes the constructor function return the newly created object IF it is not returning anything.

Long story short, when using the new operator, this inside the constructor function points to the newly created object. Keep in mind that if you forget to include new operator and just call the function directly, this will not point to the new object but rather to the global object.

var Person = function (firstName, lastName) {
  //If you called this using new operator, this links to the new object
  this.firstName=firstName;
  this.lastName=lastName;
};
var john = new Person('John', 'Doe');

In classes

When using this inside of methods of an ES6 class, it points to the current object. That's nice.

class Person {
  constructor (name) {
    this.name=name;
  }

  greet() {
    console.log('Hi! My name is' + this.name);
  }
}

var john = new Person('John Doe');
john.greet(); // Hi! My name is John Doe

However, when using nested functions inside such methods, they behave as regular functions. Same as nested functions in object methods. There is one caveat though. The code inside classed is automatically executed in strict mode even if it is not explicitly declared! As we already know, strict mode changes behavior inside functions so this is undefined instead of pointing to the global object.

In fat arrow functions

In short, fat arrow functions are a more concise way of writing function expressions introduced in ES6. Their biggest advantage is, however, not saving a few characters when typing. They greatly simplify usage of this. The thing is that unlike function expressions, where this points to the global object, in fat arrow functions this is taken from the enclosing function. Remember section about methods and how we used trick var that = this to preserve original this of the other function? Arrow functions are shorter and you don't need to worry about such tricks.

//The old way using function expressions
var john = {
    name: 'John',
    greet: function(people) {
        var that = this;
        people.forEach(function (person) {
            // this points to the global object
            console.log('Hi ' + person + '. My name is ' + that.name);
        });
    }
};

//The new ways using fat arrow functions
var john = {
    name: 'John',
    greet: function(people) {
        people.forEach(person => {
            // this points to this of the greet function!
            console.log('Hi ' + person + '. My name is ' + this.name);
        });
    }
};

john.greet(['Jane', 'James', 'Jill']); // Hi Jane. My name is John ...

Setting this directly - apply, call, bind

Sometimes it is handy to be able to directly set the value of this when calling a function. Fortunately, there are three methods in javascript exactly for that. All of them are available to all the functions as they are on the function's prototype. They are: apply, call and bind.

Apply and call are very similar to each other. They allow you to invoke their function with a value of this specified as a parameter and by providing arguments to be passed to the function call. The value returned is the value returned by the function called. The only way in which they differ is that one expects the function arguments as an array and the other one as direct arguments. Let's compare.

function myFunction(arg1, arg2, arg3) {
    console.log(this)
}

var valueOfThis = {name: 'John Doe'};

//regular function invocation, this points to the global object
myFunction('arg1', 'arg2', 'arg3');

// You can call the function and specify a value of this to be used
myFunction.apply(valueOfThis, ['arg1', 'arg2', 'arg3']); // arguments passed as an array
myFunction.call(valueOfThis, 'arg1', 'arg2', 'arg3'); // arguments passed separately

Bind is different. Bind does not invoke the function immediately. It returns function, which has this bound to the value provided. That is useful in many situations. For example, when you have a method of an object and you want to pass it somewhere else as a callback function. Without its object, it is an ordinary function and when invoked this points to the global object. You can, however, pass a version of your method where you bind your original object as this. No matter where your method ends and who will use it, you can be sure that this will have a proper value.

var john = {
    name: 'John',
    greet: function (people) {
        console.log('Hi! My name is ' + this.name);
    }
};

//When we pass a function like that, this is not preserved
var greetFunction = john.greet;
greetFunction(); // Hi! My name is undefined

//Let's bind our desired value of this using bind method
var boundGreetFunction = john.greet.bind(john);
boundGreetFunction(); // Hi! My name is John

Conclusion

Understanding how this works in Javascript under various circumstances is crucial and prevents a lot of confusion and headache. Keep in mind that this depends not on the function itself but rather on how the function is called. Since functions are first-class citizens in Javascript, they can be passed around and this will change accordingly. It is, therefore, useful to remember to prevent possible errors with actions such as:

Using strict mode to prevent accidental mutation of the global object
Binding this when passing methods around
Using arrow functions
Using classes instead of constructor functions to prevent accidental omission of the new operator

Javascript hoisting, var, let and const variables

Fri, 09 Feb 2018 22:12:03 GMT

One of the Javascript's oddities is that you can use variables and functions even before they are declared. It's called hoisting. Fortunately, in ES6 let and const variables offer much better behavior.

Hoisting

Consider the following code:

sayHi();

function sayHi() {
   console.log('Hi there!');
}

It works and prints 'Hi there!'. Even though the function is defined after it is used. Or another example:

john = 'John Doe';
console.log(john); //John Doe
var john;

Variable john is declared after it is used, yet it still works. How is this possible? When your Javascript code is being processed, in the first iteration, before actually executing it line by line, all the variable and function declarations are detected. Then they are created in memory and space is allocated for them. Only after that, the code is executed line by line. This behaves exactly the same as if the declarations were moved on top of the scope (e.g. function body). That means your code behaves as if the declarations of variables and functions were first and then the rest of the code. There is one caveat though. Consider the following line:

var john = 'John Doe';

It is actually consisting of two parts:

var john means that variable john is declared
= 'John Doe'; means that previously declared variable john is assigned a value of string 'John Doe'

It is basically one-liner for:

var john; // declaration
john = 'John Doe'; //initialization

The thing is that only declarations are hoisted, not initializations. That means you can access variable john before it is actually declared but its value will be undefined.

console.log(john); //undefined
var john = 'John Doe';
console.log(john); //'John Doe'

Another thing to watch for is that while function declarations and variable declarations using var keyword are hoisted, class declarations are not. You cannot use class before it is declared.

var john = new Person('John', 'Doe'); //ReferenceError: Person is not defined

class Person {
    constructor(name, surname) {
        this.name = name;
        this.surname = surname;
    }
}

var jane = new Person('Jane', 'Doe'); //No problems here

Best practices

It is a good idea to always declare first before using a variable or function. And to group all the declarations on the top of the scope they apply to. It is much easier to see what variables are being used in a given scope and easier to find a variable declaration when looking for it. It is also much more natural to define a construct before using it than the other way around. Such code is much easier to read, understand and to reason about. And after all, some of the developers, especially when junior or switching from other languages, may not be familiar with the hosting and it would just cause unnecessary confusion.

Block scope

If you come from a different programming language than Javascript, the following behavior may be surprising to you.

for (var i = 0; i < 10; i++) {
    // do something
}

console.log(i); // 10, What? 'i' still exists outside the for block!

Unlike in many other languages, variables declared using var keyword do not respect block scope. That means you would expect that variable 'i' from a for loop would be accessible only inside the loop. Wrong. The scope of a variable declared in Javascript depends on where you declare it.

If you declare it inside of a function body, its scope is the whole function's body. And because of hoisting, it can be accessed anywhere in the body.
If you declare your variable on the root level it is in the global scope. It is actually a variable of the global object (e.g. window).

If we take our example with for loop, the variable i is actually now declared also as a property of window - that is window.i!

for (var i = 0; i < 10; i++) {
  // do something
}

console.log(window.i)// 10, Oh the horror! 'i' is now on the global object

That is terrifying. Function wide scope is usually a lot wider than you need. Often you have variables which only are needed inside of a loop, if statement or similar block. Minimizing the scope where a variable is available is very important. It makes your functions easier to understand and reason about. Most importantly, the smaller the scope is, the smaller is the risk that some code somewhere else will mess with your variables. The bottom line is - it's useful to have smaller encapsulated scopes, rather than a big scope for the whole function. And of course, your functions should be really small anyway, which reduces the scope of the variables within and decreases complexity.

IIFE

Because it is a good practice to minimize the scope a variable is accessible in and var can be accessed in the whole function scope, there were historically some workarounds to address this. One of them is IFFE, which represents Immediately Invoked Function Expression. It is an artificial way to create a new local encapsulated scope. What you basically do is surround a block of code by a function expression and immediately call that function. Variables inside are not accessible from the surrounding code.

(function () {
    var foo = 42;
})();

console.log(foo); // ReferenceError: foo is not defined

let

In ES6 in addition to var, there were two new variable types introduced - let and const. What are they good for if there already is var? One of the differences is that their scope is not the entire function but rather only the enclosing block. That is, some section of code wrapped in curly braces {}. Like the body of an if statement or for loop. Finally, you can minimize the scope of your variables to the bare minimum without ugly workarounds such as IIFE.

Another difference (and advantage) of let over var is how hoisting works. Variables declared using var can be accessed in the whole scope (and we already know that var has a bigger scope). You can access them before they are declared. In such case, their value would be undefined though, as only declarations and not initializations are hoisted. This is generally considered a bad practice. Let behaves differently. You can access it only after it was declared. Until then, the variable is considered to be in the 'Temporal Dead Zone'.

To better understand the difference between var and let, we need to compare their life cycle. When a scope is entered, the first thing which happens is that the whole scope (function body, block) is searched for variable declarations. Then there is a memory space allocated for each of the variables. After this, the body of the function or block is executed line by line. That's basically how hoisting works and why you can access your variables before declaring them.

There is one key difference how this behaves between var and let/const though.

In case of var, after creating variable definitions, before executing line by line each of the variables is initialized with the undefined value.
In case of let/const, the initialization to undefined does not happen until the line where the declaration actually happens. And only if there is no assignment immediately. On the lines above the variable is in the Temporal Dead Zone and accessing it results in Reference Error.

One of the advantages of let over var is also pollution of global scope. When you declare a variable using var on the root level, it is automatically declared on the global object:

var foo = 42;
console.log(window.foo) //42

When using let (and const), this does not happen:

let foo = 42;
console.log(window.foo) //undefined

const

Const behaves in a very similar way to let. It is also block scoped and cannot be used until declared. There is, however, one key difference. Once variable declared using const keyword is assigned a value, you cannot reassign it. You also need to initialize the variable immediately when declaring it. It is useful as this prevents accidental reassignment of the variable. It also promotes good coding practices as it prohibits using a single variable for multiple purposes during its lifecycle, which is confusing and error-prone. And how can you even have a nice descriptive name of a variable which does multiple different things?

While this is a useful concept, you need to be aware of some limitations. The only restriction const provides is about reassignment. That does not mean object assigned to a const variable is not immutable! You can still change its properties, delete them or add new ones. You just cannot assign a completely different object. Note that applies only to objects. In case of primitives such as numbers, strings or booleans, you are safe as they are immutable.

let vs const

Because of the advantages of const described above, it is a good practice to prefer const over let. Actually, in most of the cases, you can safely use const. There are only a few cases where you really need let. One of the examples can be loops. Because the iterating variable gets changed with every step through the loop you cannot use const. However, you can avoid using loops in many cases in favor of more functional way such as - map, filter or reduce. When using a linter (and you should use it), it can check for you cases where you can safely replace let with const as the variable is never reassigned.

Conclusion

Hoisting is one of the weird Javascript's concepts, yet it is very important to be familiar with it. It is good to stick to the convention where you declare first and use later. And all your declarations should be grouped together.

To minimize hoisting problems and confusion, there are fortunately modern alternatives in ES6 - const and let. Their scope is block instead of the whole function and they cannot be used before they are declared. Because of this, you should prefer them over the good old var. Moreover, when possible the const is a safer choice over let because it cannot be reassigned.

Javascript constructor functions and new operator

Fri, 02 Feb 2018 22:12:03 GMT

What does Javascript's new operator do? What are constructor functions? Why is new criticised and what are the alternatives?

Creating objects

In your applications, most of the time you don't work with primitives but rather with objects. You can directly create objects with object literal like this:

var Person = {
    firstName: 'John',
    lastName: 'Doe'
}

This is sometimes good enough, but in many cases, you want to work with many objects, which share the same characteristics - that is: they have the same properties and methods. These can be users in your applications, bank accounts, cars or anything else. The point is, you'll want to work with 'Jane' and 'John' in the same way because they can both be considered a person - they have a name, surname, age and so on.

Alright, so how do you create objects which have the same methods and properties? Using constructor functions and the new operator.

The new operator

Let's consider the following example:

var Person = function (firstName, lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
};

var john = new Person('John', 'Doe');

The Person is a constructor function. Its whole purpose is to create new objects which represent different persons. We can be sure that each person will have the first name and the last name, which are provided as parameters to this function. Notice that unlike regular functions, constructors begin by convention with a capital letter. This way you can easily tell them apart from ordinary functions.

Then we are creating a specific person instance called john with firstName 'John' and lastName 'Doe'. Notice that there is a new keyword before the Person constructor function call. It is actually an operator and it is responsible for all the magic when creating new persons. It does actually quite a lot:

It creates a new blank object.
It makes this to point to this newly created object inside the constructor function
It sets the prototype of the newly created object to the constructor function's prototype.
It makes the constructor function return the newly created object IF it is not returning anything.

That is the reason why John is created even though the function Person does not have an explicit return statement. That's the reason why using this.firstName=firstName; sets the first name provided to the new object. Without the new keyword, this would point to the global object in our case and not to the newly created instance.

What if I forget to include new?

There are some serious problems when using the new operator. In languages such as Java, it is strictly limited how you can use the new keyword. It can be used only to invoke methods, which are explicitly marked as constructors. Constructors are guaranteed to return a new instance of the class, performing all the required initialization. Constructors can be invoked only using the new keyword and the new keyword can be used only to invoke constructors.

In javascript, the situation is unfortunately not so strict.

You can use new operator on ANY function
You can call function intended as constructor WITH or WITHOUT the new keyword

So, calling function not designed as constructor will not result in an error. What's more, you'll get different behavior based on whether you call the constructor with or without the new keyword. And it is really easy to forget to include new. After all, there is no compile-time check as in Java. Let's look at an example.

var Person = function (firstName, lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
};

var john = new Person('John', 'Doe');
var jane = Person('Jane', 'Doe');

First, there is a constructor function Person and then it is called twice. Once with new and once without. Can you guess what each variable contains?

console.log(john); // Person {firstName: 'John' , lastName: 'Doe'}
console.log(jane); // undefined
console.log(window) // Oh dear! The global object now has firstName and lastName!

What just happened? Looks like John is doing well, but poor Jane is undefined. What's worse, it looks like firstName and lastName got stored on the global object (window in case of running in a browser). So my precious new object is undefined and I am polluting the global object instead. Nasty.

The reason why is that one of the actions new operator does is changing this, so it points to the newly created object. When new is not specified, the constructor is considered an ordinary function call and this remains the same as usual. In our case, when I just declared a function on the top level and called it from the top level, this is pointing to the global object. Global object differs in each environment (browser vs server side javascript), but in a browser it is window. That means I am now storing my properties directly on the window object.

Another action that new operator does is that makes the newly created object result of the whole new expression if the function does not return object specifically. It is the case here as you can see that the Person constructor does not explicitly call return (and yet, John is successfully created). When called without new, it is just a regular function call. And functions without explicit return statement return undefined.

Protecting against forgetting to include new

Alright, forgetting the new operator can have pretty nasty consequences. So what can you do to prevent such cases?

Capital letter convention

You should always stick to the convention that constructor functions start with a capital letter and regular functions with a lowercase letter. This way you can immediately see that something is fishy when encountering a situation such as:

var person = Person('John', 'Doe'); // capital letter, I should use new

It is also helpful when constructor names are nouns and regular functions are verbs:

var size = new computeSize(); // lowercase letter;  computeSize doesn't sound like noun

Explicit check of this

Remember how one of the things new operator does is changing this to the newly created object and setting the prototype? One of the ways how to defend against the accidental omission of the new operator is checking what actually this points to in the constructor function.

var Person = function (firstName, lastName) {

    if (!(this instanceof Person)) {
        return new Person(firstName, lastName);
    }

    this.firstName = firstName;
    this.lastName = lastName;
}

This way if this is not an instance of Person, we can detect it and properly call the constructor again and make sure we add the new keyword to the call. The result is same no matter whether you use the new keyword for your constructor or not.

The disadvantage is that it encourages developers to not worry about using new as it is still safe. Building such habit is not good as not all the code you'll work with will always have such safeguards. It is much better to stick to the convention of always using new instead of just sometimes and have the usage inconsistent across the codebase. As an alternative, you may consider using the check but to throw an error to fail fast and discover improper usage of your constructors early on.

Strict mode

You can enable strict mode in your script or just a part of it using "use strict";. This basically makes some of the weird constructs result in real errors instead of silently failing or resulting in unexpected and unwanted behavior. An example can be usage of an undeclared variable.

One of the issues it addresses is pointing this to the global object (window) when not calling a function as a method of an object. When the strict mode is activated, this is undefined instead of pointing to the global object. Forgetting the new operator before a constructor call would then result in an error saying that this is undefined. You would discover your mistake early on when the strict mode is enabled. Sometimes you cannot afford to use global strict mode as it would break all your existing code. In such cases, you can always enable strict mode just for the function.

var Person = function (firstName, lastName) {
    "use strict";
    this.firstName = firstName;// TypeError: Cannot set property 'firstName' of undefined
    this.lastName = lastName;
};

Classes

In ECMAScript 6, there is a new concept of classes, which aims to replace constructor functions and addresses its issues. The person from our examples above as a class would look like this.

class Person {
    constructor(firstName, lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }
}

You can create new persons as usual:

var john = new Person('John', 'Doe');

What's really cool is that if you accidentally forget to include the new keyword when creating a new instance it results in an error instead of silently altering the global object.

var john = new Person('John', 'Doe'); // Person { firstName: 'John', lastName: 'Doe' }
var jane = Person('Jane', 'Doe'); // TypeError: Class constructor Person cannot be invoked without 'new'

What is unfortunate is that with classes and their syntax javascript gives a false impression that it uses class-based inheritance. Developers coming from other languages like Java are especially vulnerable to this. Instead, Javascript still uses prototype-based inheritance and classes are just syntactic sugar on top of good old constructor functions.

Object creation alternatives

In Javascript, there are many ways to create a new object. We already covered object literal, constructor functions and a brief mention of classes. There are more ways, which can be used as an alternative to new and constructor functions. For example, Douglas Crockford used to be a zealous advocate of not using new at all and using Object.create instead. However, later on, he changed his mind and stopped using even Object.create. Another example may be using factories instead, which have many advantages, such as more flexibility or decoupling from a specific object type. And of course, many frameworks offer convenient ways of creating objects. There is no silver bullet so it is good to be aware of different approaches and their tradeoffs.

Conclusion

When using constructor functions, make sure you don't forget the new operator. Otherwise weird things will happen. Stick to the naming conventions. The strict mode is your friend and should protect you against accidental omission. And of course, if you can use ES6, classes are a foolproof way to replace constructor function. Don't get fooled though. Even though it does sound like class-based inheritance, it is just syntactic sugar and Javascript has still prototype-based inheritance.

Javascript Primitives Guide

Wed, 31 Jan 2018 22:12:03 GMT

What are primitives in Javascript? What pitfalls to watch for? What are their object wrappers and when should you use them?

Primitives versus Objects

Javascript offers six primitive types: number, string, boolean, undefined, null and symbol. Everything else is not a primitive. It is an object. Dates, arrays, functions, regular expressions -- everything.

A simple example of an object can be a plain object created by object literal like this:

{
   firstName: 'John',
   lastName: 'Doe'
}

Examples of primitives are 61, 'Kitty' or false.

Mutability

In the example above, you can see that object is basically a collection of name-value pairs. You can change their values ('mutate' them, we say that objects are mutable) or you can even add your own new properties or remove existing ones.

var john = {
    firstName: 'John',
    lastName: 'Doe'
}

john.lastName = 'Smith';
john.age = 40;
delete john.firstName;

In contrast, the primitives are immutable, their value cannot be changed. You can, of course, reassign a variable with a new primitive, but the old one cannot be changed.

Comparison

Primitives are compared by value, objects are compared by reference. What does it mean though? If you compare two primitives having the same value, they are considered equal.

42 === 42 //true
'John' === 'John' //true

However, when comparing two objects you need to be more careful. They are compared by their identity. That means it does not matter if the two objects have same properties with the same values. They are equal only if they are the same instance. That is -- if comparing two variables, they are equal only if they point to the very same object.

var john = {name: 'John', name: 'Doe'};
var stillJohn = john;
var evilTwin = {name: 'John', name: 'Doe'};

john === john //true
john === stillJohn //true
john === evilTwin //false

You can see that two variables can point to the same object. That is because variable containing object holds only a reference to that object. When you assign the reference to a new variable, now you have two variables holding the reference to the same object. That means changing properties of the object stored in either of the variables will affect them both. This does not apply to primitives as they are not using references, their value is copied instead every time. The bottom line is -- be careful when passing your objects around -- somebody else can change them and you will be affected. For more info check this tutorial.

Primitive Types

String

Strings are basically Unicode text, 16-bits per character. You may be surprised that string is actually primitive value as it is not that common. For example, Java has primitives as well, but String is not a primitive there. Another surprise may be that there is no type representing a single character like in many other languages. You just have to simply use a string with one char instead.

Number

Unlike many other languages, JavaScript has just one type for representing numbers. That means no distinction between integer and decimal values, no distinction between signed and unsigned or single and double precision floating numbers. Javascript uses 64-bit floating point numbers.

Except for regular numeric values, there are also some special values. There is NaN (Not a Number), which is a result of operations such as 0/0. Then there is Infinity, which can be both positive and negative. You can check for minimum and maximum values fo prevent overflow/underflow - Number.MIN_VALUE and Number.MAX_VALUE. And then, of course, there is +0 and -0 in Javascript (0 is just an alias for +0).

Be careful when working with NaN as this value behaves in an unexpected way -- it is the only value in javascript that is not equal to itself, that means you cannot test that something is NaN by

something === NaN //Does not work, always false
NaN === NaN //false

You can use isNaN or Number.IsNaN (ES6+) instead of direct comparison.

Boolean

Boolean type contains just two values -- true and false.

Symbol

This is a new primitive type introduced in ES6. There is no symbol literal, so all the symbols need to be created using Symbol() function. Each symbol is unique, which makes them ideal for use as object properties to avoid clashes. Or in other cases where you need to have unique values. For more info see this post.

Null & Undefined

Both null and undefined are primitive types representing the absence of value. Null is usually explicitly returned and assigned and because of this it generally represents that the value is intentionally missing. The undefined, on the other hand, means that the value does not exist in cases, such as:

A variable was declared but not initialized
Return value of methods that do not return anything
Value of function parameters when called did not provide value
Accessing a property of an object which does not exist

One caveat to watch for and which is confusing is that even though null is a primitive, calling typeof null *[returns *object](https://stackoverflow.com/q/18808226/4560142). In contrast typeof undefined returns undefined.

Object Wrappers

Alright, we know that objects have properties, which you can access using a dot. Like person.name = 'john'. Primitives don't have anything like that, they are just a single value. And strings are primitives. How come that the following will work then?

var name = 'John';
console.log(name.length); //prints 4
console.log(name.toUpperCase()); //prints JOHN

String are not objects, you can try it by using:

typeof 'John' //string

Turns out that for certain primitives (number, boolean, string) JavaScript offers Wrapper objects, which can be used when an object is needed and provide some extra convenience methods. So what happens in the example above:

Javascript detects that you are trying to access a property of a string primitive.
It creates a wrapper object String to wrap the original string primitive.
It accesses the length and toUpperCase on the wrapper object instead of the original primitive.
It discards the wrapper object, frees the memory and continues.

This way you can use all the helpful methods on strings to make a substring, convert it to lowercase, split it and so on. Note that primitives are immutable so all these methods just return a new instance of the string and don't modify the original.

Manually creating wrappers

In the example above, JavaScript creates wrapper objects automatically under the hood. There is also a way to create such objects explicitly.

new Number(4)
new String('Hi')
new Boolean(true)

These are not primitives anymore, but objects, which contain the primitive value and add some extra goodies.

typeof 4 //number
typeof 'Hi' //string
typeof true //boolean

typeof new Number(4) //object
typeof new String('Hi') //object
typeof new Boolean(true) //object

This has some serious implications. You cannot really compare wrapper objects and primitives:

'Hi' === new String('Hi') //false

Also, be aware that objects are considered truthy, that means that you need to watch for this:

if (false) {
    //This does not execute
}

if (new Boolean(false)) {
    //This executes
}

Because of this, explicitly creating wrappers using the new operator is considered a bad practice. On the other hand, calling the function without the new operator is perfectly valid as it just tries to convert the input into the corresponding primitive type and returns a primitive value.

typeof new Number('42') //object, not recommended
typeof Number('42') //number, safe

valueOf and toString

When you are working with objects, there are some cases when you would rather need a primitive representation of the object. Actually, javascript offers two methods for it which each object inherits:

valueOf() returns primitive representation of the object
toString() returns string representation of the object

A good example can be Date. toString() returns a human-readable description of the date, while valueOf() returns a number representing the date as the number of passed milliseconds since 1 January 1970 00:00:00 UTC and the date.

var date = new Date();
console.log(date.toString()); //Wed Jan 24 2018 14:12:07 GMT+0100 (Central Europe Standard Time)
console.log(date.valueOf()); //1516817671281

You rarely need to call valueOf() yourself, but javascript does it under the hood when a primitive is expected, for example when using + operator. You can even define your own implementation of valueOf() and toString():

var john = {
    firstName: 'John',
    lastName: 'Doe',
    age: 45,
    valueOf: function () {
        return this.age;
    },
    toString: function () {
        return `${this.firstName} ${this.lastName}, ${this.age}`
    }
};

console.log(john.toString()); //John Doe, 45
console.log(john + 1); //46

Summary

Javascript provides six primitive types -- number, boolean, string, null, undefined and symbol. Primitives are immutable and are compared by value. When needed, Javascript wraps a primitive by an object wrapper. It is not recommended to create these wrappers explicitly. If the conversion needs to be done the other way around -- from an object to a prototype, the valueOf method is called to obtain a primitive value. You can use your own implementation of valueOf.

IntelliJ IDEA Tips & Tricks: Synchronization and Sharing of Settings

Wed, 24 Jan 2018 22:12:03 GMT

Did you know IDEA offers two ways of synchronization of you settings among multiple instances of your IDE? You can use it when working on multiple computers or to share unified settings in your team.

The importance of synchronization

As a developer, you usually use multiple computers each with its own instance of your favorite IDE. It can be quite a hassle to keep settings the same on all of them. Rather than doing manually, it is much better to use automated synchronization. Setting up a new IDE is much easier this way and everything is also backed up online in the process.

And of course, you usually do not work alone but in a team. It is important to stick to some guidelines and standards for your project. And having same key settings of your IDE will help a lot. Not to mention that it makes onboarding a new developer and making them stick to conventions much easier.

Settings Repositories

The first way to synchronize settings is using Settings Repositories. The way it works is that it uses an external Git repository to share the settings. Each IDEA instance then connects to this repository to fetch the settings. In case one instance makes changes in the settings it can be propagated back to the repository. What's nice is that it is a plain old Git repo. That means your settings are backed up and under version control. So you can track and rollback the changes if necessary or even allow other people to create Pull Requests with some tweaks to your settings.

To enable this feature just go to:

File → Settings Repository

And enter the URL of your git repo. If you are just starting an empty repo will do. When you are configuring Settings Repository this way, you can always check whether your local settings should overwrite those on the server, vice versa or if they should be merged.

And that's it, you're done. In most cases. However, IDEA offers a neat feature -- you can actually define multiple Settings Repositories. What is it good for? Well, maybe there are some settings you want to share with your teammates to ensure consistency on your project. But then, of course, you want some of your setting to be personal and just for you. After all, every user has different preferences.

In such case, IDEA will consider one of the repositories primary. That is the one for your own settings. The synchronization works both ways -- from the server to the client and the opposite way. The rest of the repositories are secondary and are read-only. You can only fetch settings from them but they are not affected by your local settings changes.

To configure multiple repositories go to

Settings → Tools → Settings Repository

Here you can see your primary repository, which you set up earlier. Underneath you can configure additional secondary repositories:

Note that you can also disable automatic synchronization and use just sync on demand by VCS → Sync Settings.

Sync settings using JetBrains Account

The second way of settings synchronization was available for some time as a standalone optional IDE plugin (which required an invitation to use), but now is integrated and enabled by default since 2017.3. It utilizes your JetBrains account to synchronize the data.

Under the hood, it actually uses a variation of Settings Repository mechanism, but you are shielded from it. It is much more convenient because you don't have to create a Git Repo to store the settings and provide its address to your IDE. On top of that, unlike Settings Repositories, it allows you to also synchronize your installed plugins. Pretty cool, right?

The first thing you need to do is to make sure you have a JetBrains account. If you don't have one, you can create it here. Once you have your JetBrains account, you'll need to Log in to IDEA using it. Just head to Help → Register. Then you just need to select Activate new License with JetBrains Account and enter your credentials. Then click Activate.

Before you link your IDE to JetBrains account, your status bar in the lower-right part of the window should look something like this:

After your JetBrains account is linked, you should see a new Gears icon which is used to manage Settings synchronization:

If you click it, you'll be able to enable the settings synchronization:

When you enable it on another machine, you'll be prompted how to manage settings conflict. You can either select to overwrite your local settings with the setting from the JetBrains account or vice versa.

Installed plugin synchronization is handled separately from the rest of the settings. You can either manually trigger plugin sync or enable silent automatic synchronization of your plugins in the background.

If you prefer manual synchronization, you can have fine-grained control of what plugins are synchronized and how. That means you can decide for individual plugins and you can even force installation or uninstallation of plugins or your other instances of IDEA.

And of course, if you no longer want to keep certain IDE instance synchronized, you can disable the synchronization using the gears icon in the status bar.

Conclusion

Being able to synchronize your Settings among multiple IDE instances is very important and handy feature. IntelliJ IDEA now supports two different ways of synchronization out of the box.

The first one is using Settings Repositories. It allows you to define multiple Version Control Repositories, which will act as a storage for your settings. It is useful when you want to share the settings in your team. Because you can specify multiple repositories, some of the general settings can be shared with your team members while you can still have your personal settings on top of that synced among all your computers.

The second one uses your JetBrains account to store your settings. Because of it, it is more suitable just for your own personal settings as you are not able to share your JetBrains account with your colleagues. On the other hand, it is able to manage plugin synchronization.

In either case, it is useful to have your settings consistent and backed up in a remote repository.

Web Page Audit with Chrome Developer Tools and Lighthouse

Mon, 22 Jan 2018 22:12:03 GMT

Did you know Chrome can perform a comprehensive audit of a web page and analyze many different categories such as Performance, SEO, Best Practices, Accessibility, Progressive Web Application compliance and more?

Running an Audit

In version 60, Chrome introduced a great new addition to its Developer Tools. It is a comprehensive Audit feature for web pages powered by a tool called Lighthouse. It is an open source tool available on GitHub. To run an audit, you need to open Chrome Dev Tools (F12) and select Audit tab:

Then you click the Perform an Audit button and select the categories in which you want to perform an analysis. Then just click Run Audit:

Chrome now analyzes your page for various potential problems in the selected categories. After the audit is done, you receive a score in each of the categories up to 100 points:

Lighthouse CLI and more tools

Google chrome extension

Originally, before Lighthouse was integrated as a part of Chrome Dev Tools, it was available as a standalone Chrome extension. And it still is. This may be handy if you are stuck with an older version of Chrome. Or If you want to try the newest features of Lighthouse. That's because stable Chrome does not include the latest and Greatest version of Lighthouse, but the extension does.

Command Line Interface

Having Lighthouse integration directly in Chrome is neat, however, it has its limitations. You are likely to run the audit several times when it is still new and shiny or during initial development. Or when you remember to do so. But there is a constant risk that new changes will impact the performance or violate some best practices. It is much better to integrate the audit as a part of your build process to automate the whole audit thing.

Fortunately, Lighthouse is available as a Command Line utility distributed as NPM package. It is also handy for cases where you need a more fine-grained configuration as it offers greater customization than Dev Tools currently. To make things even easier for your build, it offers a Webpack Plugin.

Lighthouse as a Service

If you want lighthouse integration even simpler, there is a tool called Treo. It basically is Lighthouse as a Service. It offers a nice GitHub integration as well as means of breaking the build in case the expectations are not met.

Chrome Canary

Note that at the time of writing the latest version of chrome, 63, includes Lighthouse 2.5.1, which does not include some of the features described in this post. For example the whole SEO category. This post was written based on Chrome Canary 66, which includes Lighthouse 2.8.0. Chrome Canary includes all the new features which will stable release contain in the future but are not yet fully tested, stable or even finalized.

Conclusion

Lighthouse offers a nice way to test your web app for performance issues, PWA compliance, SEO, Accessibility and other best practices. Even though the tool is integrated nicely with Chrome Developer Tools, you may want to consider integrating it with your build process to continuously check for possible problems.

How to protect your HTTP Cookies

Thu, 18 Jan 2018 22:12:03 GMT

Protecting your sensitive cookies is very important as stolen session cookie means an attacker can take your identity and wreak havoc.

What are Cookies

HTTP protocol itself is stateless. That means when a server receives two requests it cannot tell whether they originate from the same user. There is no conversation context preserved between requests. While it has many advantages such as good scalability, often you want to identify a user you already interacted with and keep state. For example, a user should usually be able to log-in and after that, all their requests should be considered as those of an authenticated user. In other words, you want to recognize such user when they send their next request.

But how can you achieve this? One way is using Cookies. Whenever a server receives a request from a client, it can provide a special HTTP response header called Set-Cookie. It can look something like this:

Set-Cookie: [cookie_name]=[cookie_value]

What it basically says is: Here is a little piece of information (cookies are key-value pairs). I want you to remember it. And I want you to send it back to me with your future requests. So I know it is you. The client's browser receives this information and stores the cookie on the client's machine. Whenever that client makes a request to the server, the browser checks all the cookies, selects these which are relevant to the current server and sends them as HTTP headers.

Cookie:=[cookie_name]=[cookie_value]; [other_cookie_name]=[other_cookie_value]

Securing the cookies

Okay, so cookies can be used to store information on the client and then sent to a server with each request. Information such as the unique identifier of a logged in user. And it is sent in the form of HTTP header. Now imagine someone gets their dirty hands on your precious cookie. Since your session cookie represents your identity, the attacker can use it to impersonate you. To perform restricted actions as if they were you. Transfer money, steal sensitive information, delete important data -- this kind of stuff. This means you need to make sure your cookies are as protected as possible. So what are the ways you can mitigate the risk of someone stealing your cookies?

Preventing Cross Site Scripting with HttpOnly attribute

Cookies can be directly accessed from javascript. It is very easy:

var cookies = document.cookie;

Not only it is easy to steal cookies this way, but the attacker can tamper with existing cookies or create new ones:

document.cookie = "my_precious_cookie=some_harmful_value";

You may think you are safe because you are in control of what javascript gets executed on your pages. Wrong. If your application contains Cross-Site Scripting vulnerability, it is easy for an attacker to inject any malicious javascript to be executed on the victim's machines. Once they do that, it is easy to steal your cookies and send them to the attacker.

Because of such situations, it would be really handy to disable access to your cookies from javascript. Fortunately, you can do it quite easily. When the server sets cookies, you can provide some additional attributes. One of them is HttpOnly, which says that the cookie will not be accessible in the client's browser from javascript:

Set-Cookie: [cookie_name]=[cookie_value]; HttpOnly

Protect Cookies during transport with Secure attribute

Alright, we've mitigated the risk that the attacker will steal or change our cookies from javascript. But there are more ways to steal our cookies. One of them is in-transit. That means someone can read the request on its way to the server and they will be able to see the cookie as it is just an HTTP header. How do we prevent this?

You should use HTTPS of course, so your data is secure even in-transit. This way man-in-the-middle cannot read your cookies. Unless... Well, there are ways a client can access your HTTPS site using regular HTTP. Maybe they manually type http://. Maybe you have mixed content. Maybe you do use HTTP Strict Transport Security to enforce HTTPS, but the client's browser does not support it yet. Either way, you risk your precious cookies transmitted over an unsecured channel.

Fortunately, there is a way to prevent this. Similar to HttpOnly, there is another attribute that instructs the browser that it should only send the cookie over a secured channel.

Set-Cookie: [cookie_name]=[cookie_value]; Secure

Prevent Cross-Site Request Forgery with SameSite attribute

When a browser reads a page, there are usually some resources from other domains, which are loaded as well such as images, scripts or social media buttons. The default browser behavior is that it sends cookies which browser has for the external site with the request. Since cookies are sent with cross-domain requests, this can be exploited to trick a previously authenticated user into performing some restricted action on the external site. This type of attack is called Cross-Site Request Forgery (or CSRF).

Fortunately, when setting a cookie, you can specify that browser should send it to you only if the request is originating from the same origin (is not cross-domain). This cookie parameter is called SameSite. It has two possible values:

strict
lax

Strict option prevents sending cookies from different origin altogether. While this is safe, it can be really painful for the users. For example, if you click a link to a social media or GitHub, you would be redirected there without being logged in and you would need to authenticate again.

The second option, lax, is more forgiving. It will allow sending the cookie cross-origin as long as the HTTP method is GET only and you are navigating to the root (top level).

While this is really good protection against some sorts of CSRF (still does not help if the session ID is, for example, transferred a URL parameter), it is not yet widely supported by the browsers (as of 1/2018). You can check current support on Can I Use.

Minimize Cookie Availability

Don't Share cookies with subdomains

By default, when not specified otherwise, a cookie is only sent to a sub-domain, which set the cookie. That means that if you set a cookie from account.example.com it will not be sent to forum.example.com. You can, however, specify the Domain attribute of the cookie.

Set-Cookie: [cookie_name]=[cookie_value]; Domain=account.example.com

If you specify just a root domain (example.com) and not the subdomain (account.example.com), the cookie will be sent to ANY subdomain of the root domain. Don't do this as you want to minimize the number of places your cookies are sent to.

Don't share cookies with other applications

There is another parameter which can be used to define the scope of the cookie, similar to Domain. It is called Path. It is used to define resources under the domain, to which the cookie will be sent. For example, path '/' means everything, /blog means every resource starting with 'blog', eg:

http://www.example.com/blog/2017/

By default, if unspecified, it is set to the path of the resource which set the cookie. Be careful not to set manually a value which is too permissive. When you set '/' that means that all the resources on the given domain will receive the cookie. This can be dangerous when you have multiple applications sitting on the same domain which are different only in their path:

http://www.example.com/first-app
http://www.example.com/second-app

In this case, the apps would be sharing cookies. That means that if one of them is vulnerable it can make the other one vulnerable as well as you could steal the other app's cookies. For example using Cross Site Scripting.

Limit cookie lifespan

In the same way you want to reduce the cookie's scope to minimize the number of places where it can be stolen, you want to minimize the cookie's lifespan to decrease the time when the cookie is vulnerable.

Cookie lifespan is, if not specified, just for one session. That means until the user closes their browser. Well, at least in theory. These days browsers usually have some mechanisms for session restoration or continue to run in the background so you cannot depend on session cookies being immediately wiped.

You can specify cookie expiration or lifespan using Max-Age and Expires attributes.

For sensitive cookies use the shortest sufficient lifespan possible. Using these parameters you can set cookie lifespan to infinity, which is obviously not a good idea. It is also worth mentioning that to further limit the lifespan of your session cookies, you should limit session duration server side and provide your users with a means of manually terminating their session (logout).

Conclusion

It is extremely important to protect your sensitive cookies, such as these containing session identifiers. If such a cookie is lost, an attacker can assume your identity and do everything you have permissions to do. You should protect your cookies against Cross Site Scripting with HttpOnly attribute so they are not available from javascript.

Cookies are also vulnerable during transport, so you should apply HTTPS and make sure your sensitive cookies are not transmitted using plain HTTP by specifying Secured cookie attribute.

To defend against Cros Site Request Forgery, you should add the SameSite attribute with either strict or lax value. But be aware of currently limited browser support.

The limit attack vectors you should keep the availability of your cookie at the minimum. That means limiting the scope (Domain and Path) and lifespan (Expires and Max-Age).

IntelliJ IDEA Tips & Tricks: Testing REST services with integrated HTTP Client

Wed, 17 Jan 2018 22:12:03 GMT

It is often useful to be able to send HTTP requests directly from your IDE. IntelliJ Idea offers full-fledged HTTP client which you can use without the need of a dedicated external client.

Integrated HTTP Client

You often need to call an HTTP endpoint and examine the response received. In more complicated scenarios you need to use various HTTP methods, send HTTP Headers or cookies. While there are many dedicated programs just for this, IDEA offers this feature out of the box, which is much more convenient. The IDE actually offers two main ways to do so.

The Old Way - Test RESTFul Web Service

This feature has been in IDEA for quite some time. You can access the REST client by going to:

 Tools → HTTP Client → Test Restful Web Service

This will open a new tool window called REST client.

The usage is pretty straightforward and similar to many other HTTP clients. First, you need to specify an HTTP method (GET/POST/PUT/...), target URL and path. You can provide additional data on the Request tab, such as HTTP headers, request parameters, and body. You can also specify cookies to be sent on a separate Cookies tab.

And then you just hit Submit Request. IDEA sends a new HTTP request to the target location and displays the response received in the Response tab. Response headers can be viewed in the Response Headers tab.

And that's pretty much it. On top of this, the client offers some additional features such as:

Importing and exporting request from/to an XML file. This can be handy when reporting bugs, so the issue can be easily reproduced. And also to persist request between starts of IDEA as REST client's state is not preserved.
You can generate basic authentication header from a username and password.
Request/response history is saved.
Integration with JAX-RS - IDEA can detect your @Path annotations and offer them in the path field, so you don't have to manually search for the value.
Code completion for common HTTP headers.

The New Way - Using Http Request File

Even though the REST client is no doubt useful, it has some shortcomings. It is quite cumbersome to add values such as cookies or HTTP headers. You have to do a lot of clicking and moving around before you create a request. In IntelliJ IDEA 2017.3, there is a new way to send HTTP requests, which is simpler and yet more powerful. It is HTTP Request file and it uses HTTP client directly in the Editor window.

Getting started

The easiest way to start is just to add a file with .http extension to your project. The name does not matter, just the extension. Alternatively, you can right-click a directory and select New → HTTP Request. This file will be a source file for the new Editor-based HTTP client. Since it is an ordinary file, it will persist between IDE starts, no need to export/import XML files as before. Moreover, it can be shared with other devs or committed to version control.

Sometimes it can be handy to have the requests stored permanently. In other cases not. If you need just a quick test and create one or two throwaway requests, which will not be reused, it is much better to use a scratch file. Just press Ctrl + Shift + Alt + Insert to open the New Scratch file dialog. Then select HTTP Request.

No matter which variant you use, the usage will be the same, only the persistence of the file is different.

A simple request

The simplest thing you can do is just to type a URL and click Run icon in the gutter. Or alternatively, Alt + Enter and Run.

When no HTTP method is specified, it assumes you want to use GET. No headers or parameters are sent. Then the response is displayed:

HTTP Methods and headers

For simple GET requests, you are already covered. But often you need something more complicated. First of all, you will need other methods than just GET. To specify an HTTP method, simply add its name before the URL:

GET http://localhost:8090/persons/all

Then you'll need a bunch of HTTP headers. Just add them below the URL, one per line as key-value pairs separated by a colon. Since cookies are just HTTP header, you can include them with the other headers.

GET http://localhost:8090/persons/all
Accept-Language:en-US,en;q=0.9
Authorization:BasicQWxhZGRpbjpvcGVuIHNlc2FtZQ==
Cookie: name=value; name2=value2; name3=value3

Adding body

But HTTP headers are not enough. For methods such as POST, you'll also want to specify body the of the request. You can do so by leaving one blank line after the end of the request declaration. The body can be multiline. You can have multiple requests in one file and separate them by a line with ###.

POST http://localhost:8090/persons/add/
Content-Type: application/json

{
    "name": "JohnDoe",
    "age": 29
}

Alternatively, you can specify, that the body should be read from an external file. To do this, you need to provide left angle bracket followed by space and the path to the file.

POST http://localhost:8090/persons/add/
Content-Type: application/json

< person.json

Note that IDEA will help you with autocomplete for both existing files for body and HTTP Headers. What's more, if you specify content type and the body of the request, IDEA automatically treats the request body as if it was written in that language. That means if you choose XML for example, it will check validity, autocomplete tags for you and all the other language-specific features available in the editor. Also, you can include comments in the file using #. It is useful for documenting some special HTTP headers or parameters for example.

Environmental variables

Your application is usually deployed in multiple environments. You start developing locally, then it gets deployed to Development Environment, UAT and eventually Production. The names and number of the environments can vary, but what is certain is there is more than one. When using HTTP client, the requests are usually almost identical among the environments, except for several environment-specific variables, such as URL.

IDEA allows you to reuse the same requests among multiple environments by specifying environmental variables and then using placeholders for the variables. Then it is really easy to switch between environments.

The environments and their variables are defined in a file called rest-client.env.json. It is not supplied by default, so you need to create one. Bellow is the example of three environments with different host variable.

{
   "local":{
      "host":"localhost"
   },
   "development":{
      "host":"dev.example.com"
   },
   "production":{
      "host":"example.com"
   }
}

In your HTTP Requests file (even scratch) you can then reference variables surrounded by double curly braces:

GET http://{{host}}:8090/persons/all/

You can run specific configuration by right-clicking the run icon in the gutter:

HTTP Proxy

In case you need HTTP proxy for your calls, the client shared proxy settings of the whole IDE, which can be configured under Appearance & Behavior → System Settings → HTTP Proxy.

History

What's cool is that IDEA automatically stores the history of all the requests you performed (up to 50 items). It is available from Tools → Show HTTP Requests history. What it actually does is that it opens a file called http-requests-log.http in .idea/httpRequests folder. It can look something like this:

POST http://localhost:8090/persons/add
Content-Type: application/json

< C:/_MY/rest-demo/person.json

<> 2018-01-11T032255.200.txt

###

GET http://localhost:8090/persons/all

<> 2018-01-11T115612.200.json

###

The really nice thing is that it is just another .http file. That means that you can directly execute the requests from there again and it uses the same syntax. The second cool thing is that it preserves not only requests but also responses (response bodies only, unfortunately). Each Response body is saved in a separated file indicated by angle brackets. The type of the file corresponds to the received content type, so here it is JSON, but it can be HTML or something else. You can navigate to the response file by CTRL+Click. Actually, the same response information is also shown in scratch files, but not in physical .http files. When performing the same request multiple times, the response info is preserved for each invocation. Then you can easily compare two of the responses by IDEA's integrated DIFF tool.

It is nice that the same history and log are used both for the new editor based HTTP Request files and the old REST Client, which runs in a separate tool window.

Conclusion

IDEA's integrated HTTP client capabilities are now stronger than ever with the new HTTP Request files. What's so cool about them is the fact that it is not an HTTP client in a separate tool window with various tabs, text-fields, and buttons, but everything happens directly in the Editor window. That means you get all the goodies such as autocomplete or language injection. And it is much simpler and quicker to use as you don't have to navigate a complicated UI. That means that it is much easier to use it with keyboard only also. This is the state as of IDEA 2017.3, according to the JetBrains, we have many more features to look forward to in the future releases.

Java 10 – var – Local Variable Type Inference (JEP-286)

Thu, 04 Jan 2018 22:12:03 GMT

In Java 10, you no longer need to explicitly declare a type of local variables, which significantly reduces boilerplate and increases readability.

Type inference in Java 7 - The Diamond operator

There is no denying that Java can be very verbose. Take variable declaration and assignment for example.

User joe = new User();

The type is repeated twice even if it is the same on left and right. With Generics, the boilerplate is even more apparent.

Map<User, List> accounts = new Map<User, List>();

In Java 7, there is a nice feature, called diamond operator. It allows you to provide just a pair of empty angle brackets instead of all the type parameters declaration on the right-hand 's side.

Map<User, List> accounts = new HashMap<>();

Much better, right?

JEP 286: Local-Variable Type Inference

Even though the diamond operator helps a bit, you can use it for generics only, and there is still a lot of boilerplate. What if you could just write 'var' instead of an explicit type like this:

var joe = new User();

No more type duplication and much more concise, right? Don't get fooled though - this does not mean that Java has suddenly a dynamic typing now. Everything is still statically typed. It is just syntactic sugar for writing:

User joe = new User();

How is this possible? Well, if you are declaring a variable and initializing it at the same time (assigning a value), you can detect what type your variable should be based on the value assigned. No need to repeat the type twice. Since it is just a syntactic sugar, the compiler detects the type at compile time and the resulting bytecode is exactly the same as if you declared the type explicitly.

This feature is likely to be part of Java 10 and is described in detail in Java Enhancement Proposal 286.

Better readability

The main benefit of this feature is not that you can WRITE less code. Optimizations of writing speed are greatly overrated and are much less important than reading speed and readability. After all, you are going to spend much more time reading a piece of code than writing it.

Using var improves the readability in two ways. First, there is a lesser amount of text to read and the information about type is not duplicated. That means less boilerplate to filter out when reading. Second, when all the type definitions on the left have the same length, it is much easier to read line by line as all the variable names are neatly aligned. And when you are using nice, self-explanatory variable names clearly revealing intent, it is in most cases all you need to know. Type is just a detail, which you can look for when needed, but unnecessary when skipping through the code. Compare this:

var userService = new UserService();
var userAccounts = userService.getUserAccounts();
var myAccounts = userAccounts.get(ME);

To This:

UserService userService = new UserService();
Map<User, List<PersonalAccount>> userAccounts = userService.getUserAccounts();
List<PersonalAccount> myAccounts = userAccounts.get(ME);

Limitations

Locations and situations where you can use var are actually somewhat limited. As the name suggests, you can use it only for local variables. Not for class fields, not for method return types. The only place where you can use them is in for and for-each loops:

for (var user: users) {...}
for (var i = 0; i < users.size(); i++) {...}

Another limitation is that variable declared using var needs to be immediately assigned. That means the following is not valid:

var joe;
joe = new User();

Initializing var variable with null is also not allowed. You cannot also declare multiple variables in a compound declaration.

int x = 1, y = 2; //OK
var x = 1, y = 2; //Not OK

Also, be careful when combining var and the diamond operator.

var users = new HashHap<>();

In this case, you are forcing the compiler to infer both left and right-hand side. Usually, it will result in a compiler error. Or in some cases, it will just infer Object as a generic type parameter.

Backwards compatibility

How does this feature affect your existing applications? Will they be compatible? Adding a new keyword would mean that you no longer can use it as an identifier, which would break existing applications. In fact, var is not a keyword, but rather a reserver type name. That means you can still use var as variable, package or method name. You cannot, however, use it now as a class or interface name. Applications doing so will no longer compile. Fortunately, such name conflicts with Java naming convention, so you should be safe. As long as you follow the convention, that is.

Performance

If you are concerned about a performance hit this feature could bring, don't be. Remember, the types are not inferred at runtime but at compile time. That means the resulting bytecode is the same as with explicit type declaration - it does include the information about the type. That means no extra processing at runtime.

Criticism

Readability decreased

Some argue that using var achieves actually the opposite effect - the code is less readable as in some cases it is not obvious which type a variable has when reading the code. That is mostly in cases when not explicitly calling a constructor in the assignment but rather calling a method. To some degree, it is true. You need to be careful in such cases. The good news is that you can still use explicit type declaration when appropriate. But usually, it may not even be necessary. When you have a good name of the variable and the method called, you will have a pretty good idea what is going on even without explicit type. If you find a cryptic statement with var, it will be in many cases just an indication that you cannot express intent in variable and method names well enough. And of course, a good IDE will help a lot.

val not included

In most of other statically typed languages, the Local Variable Inference is already a thing. Java is, as always, late to the party. On JVM, both Kotlin and Scala support it and both in a different way than Java. Instead of just var, they provide two options - var and val. Var is meant for variables, which can be reassigned, val for those that not (it is the equivalent of Java's final keyword).

Why does not Java offer the same? Actually, when the JEP-286 was discussed, there were both variants on the table. There was a survey with about 2500 respondents, which was supposed to gather preferences on which option should be chosen. Including some other naming conventions such as let or auto.

While var + val variant was a clear winner by a number of votes (48% vs 23%), it was rejected in favor of var only. To quote Brian Goetz:

While it was not the most popular choice in the survey, it was clearly the choice that the most people were OK with. Many hated var/val; others hated var/let. Almost no one hated var-only.

There are more reasons of course. One of them is a clash with final keyword. Val would basically mean final var leading to inconsistency of final keyword usage. For local variables, val would not be so important anyway as they are usually effectively final anyway and immutability is much more important on field level than on method level, where each method is ideally very short.

Try It

Even though Java 10 is not released yet, you can already try the feature. Download JDK 10. IntelliJ IDEA already has good support for this feature.

Updated: Style guide

Check out this useful guide for 'var' best practices - Style Guidelines for Local Variable Type Inference in Java.

Updated: Java 11

Currently (even pre-java 10), for lambdas you can omit explicit type declaration and it will be inferred.

(x, y) -> x.process(y)

From Java 11, you'll be able to use var here also, so the syntax is uniform.

(var x, var y) -> x.process(y)

See JEP 323: Local-Variable Syntax for Lambda Parameters for more details.

Conclusion

The Local Variable Inference is a nice feature, which reduces boilerplate and increases readability. As always, var on its own is no silver bullet. You need to make sure that you use descriptive variable names and in case type information is not apparent you can still use good old explicit type declaration.

OWASP Top Ten 2017

Mon, 01 Jan 2018 22:12:03 GMT

OWASP released 2017 version of top 10 Web Application Security Vulnerabilities. What are they, what is new and is your app vulnerable?

What is OWASP?

OWASP Stands for Open Web Application Security Project. It is a non-profit organization dedicated to improving the security of software, raising awareness of security issues and providing tools and resources to mitigate security risks. It consists of numerous projects with various focuses. Examples of such projects are OWASP Application Security Verification Standard Project, OWASP Testing Guide or OWASP Dependency-Check.

OWASP Top Ten

OWASP Top Ten is one of the OWASP projects, probably the most famous one. It is a list of Top 10 most critical web application security risks. It is not an exhaustive guide (there are other OWASP projects for that), but a rather short document, where each vulnerability is described on just one page. The intent is, therefore, to raise awareness about security vulnerabilities rather than provide a comprehensive description. This makes it a good starting point when you want to start taking your application's security more seriously and also a nice 'executive summary' material when you need to convince the management, that you need to allocate some of your resources to security reinforcement.

OWASP Top Ten is released periodically every few years. The most recent version was released in December of 2017. The previous versions were 2013, 2010 and 2007.

Top Ten 2017, the first attempt

The first attempt to release OWASP Top Ten 2017 was in April of 2017. It was heavily criticized and the release was rescheduled after the first Release Candidate. What was wrong?

The changes were basically:

One Item removed
Two similar items merged
Two new items added

The problem was mainly with the two new items:

Insufficient attack protection
Unprotected APIs

First of all, the two items were not added based on the actual data gathered. They were not a result of a process and methodology applied to the other eight items. They were chosen just based on the judgment of the authors. This gives you a weird mix of items, where some of them are added based on a methodology and data and some not. With no clear indication of which is which. Moreover, the independence of the project was questioned as one of the items was added based on a suggestion of a commercial company (and only on a suggestion of that company), which by coincidence offers a product addressing the issue. What's more, it was specifically mentioned in the OWASP Top Ten Document. The two new items were also rather general and not a specific exploitable vulnerability on the same level such as XSS or CSRF. The type and granularity of the ten items were therefore not consistent.

Top Ten 2017, take two

After the widespread criticism of the first version, OWASP took the feedback seriously and made a lot of changes. First of all, there was a change in the leadership. Then the methodology. Everything is now available on GitHub - The document, issues, feedback, and even the data gathered based on which the top ten issues are chosen. What a refreshing change compared to mailing lists.

Only 8 of 10 items are now chosen based on vulnerability data gathered from companies. The other two items are chosen based on a community survey. This way OWASP Top Ten contains both vulnerabilities based on current hard data as well as forward-looking items based on expert opinion of the community. The order of the items in the list is now determined by the risk factor rather than any other means.

After the first attempt failed, the new and shiny version of OWASP Top Ten 2017 was finally released in December 2017. It consists of the following ten items:

Item	Description
1. Injection	Injection flaws, such as SQL injection, occur when untrusted data is sent to an interpreter as part of a command or query. It can trick the interpreter into executing unintended commands or accessing data.
2. Broken Authentication	Application functions related to authentication and session management are often implemented incorrectly, allowing attackers to assume other users' identities.
3. Sensitive Data Exposure	Many web applications and APIs do not properly protect sensitive data. Attackers may steal or modify such weakly protected data. Sensitive data may be compromised without extra protection, such as encryption at rest or in transit.
4. XML External Entities (XXE)	External entities in XML can be used to disclose internal files, remote code execution or DDoS attacks.
5. Broken Access Control	Attackers can exploit access restriction flaws to access unauthorized functionality and data.
6. Security Misconfiguration	Use of insecure default configurations, incomplete or ad hoc configurations, open cloud storage, misconfigured HTTP headers, and verbose error messages containing sensitive information.
7. Cross-Site Scripting (XSS)	Applications use unsanitized user-supplied data in a web page. It allows execution of scripts in the victim's browser.
8. Insecure Deserialization	Insecure deserialization often leads to remote code execution, replay attacks, injection attacks, and privilege escalation attacks.
9. Using Components with Known Vulnerabilities	Components, such as libraries, frameworks run with the same privileges as the application. Components with known vulnerabilities may undermine application defenses.
10. Insufficient Logging & Monitoring	Insufficient logging and monitoring, with missing incident response, prevents rapid reaction and allows continuous probing for vulnerabilities.

What changed

Cross-side request forgery removed

CSRF is a type of attack where an unsuspected authenticated user is tricked into performing restricted actions.

Removing this is actually quite a big moment as CSRF was one of the evergreens in OWASP Top Ten. When it was included for the first time, basically no application was protected as the vulnerability was completely new. These days, fortunately, it is a wide-known vulnerability and many frameworks provide CSRF protection by including special CSRF tokens out of the box. Most of the applications are therefore safe even if the developers have no clue what CSRF actually is. According to the data gathered only about 5% of the apps were vulnerable.

Unvalidated redirects and forwards removed

This vulnerability uses redirect and forward mechanisms of trusted web applications to transfer users from trusted websites to malicious ones.

This vulnerability still affects 8% of the applications, but it was edged out from the list by XXE.

Insecure Direct Object References merged with Missing function-level access control

The two items are no longer separate issues but are now merged into one item called Broken Access control.

New Item: XML External Entities

This is the only new item which is based on actual vulnerability data gathered and not from a community survey.

The problem is that unlike Cross Site Scripting, Injection or Cross Site Request Forgery, this vulnerability is relatively unknown. Many of the current security tests do not take XXE into consideration. At the same time, consequences of a successful attack can be severe.

XXE is a type of vulnerability, which usually occurs when processing XML using outdated or misconfigured XML processors. This vulnerability can be exploited in many ways including, but not limited to, remote code execution, denial of service, port scanning or sensitive data disclosure. To mitigate the risk, you should consider using other data formats such as JSON where possible. Always use up to date version of XML parsers and processors. Where possible turn off processing of XML external entities. Unfortunately, it is usually enabled by default and needs to be explicitly disabled. Consider validating XML content server-side using a whitelist of possible values when external entities are necessary. For more details see OWASP XXE Prevention cheat-sheet.

New Item: Insufficient Logging and Monitoring

This item was included based on community survey as one of the two items selected this way. Community-selected items were included for the first time in 2017.

When an attacker is trying to exploit a vulnerability to perform a successful attack, they have to know about the vulnerability in the first place. Because of this, there is usually a lot of probing preceding the attack searching for common vulnerabilities. All the applications contain some kind of vulnerability and it is only a matter of time until it is found and exploited. When there is no detection of such probing, it can freely continue without any countermeasure. If the attack is successful, there needs to be some kind of intrusion detection mechanism, which will inform you that you were targeted by an attack. If you detect this early, you can usually prevent further damage. Sadly, the average time until such breach is detected is 191 days, which gives the attacker plenty of time to wreak havoc.

The bottom line is - this vulnerability makes all your other vulnerabilities much more exploitable and makes it hard to do a rapid response after an attack. Make sure you don't neglect logging and all your logs are easily available. It is worth having automatic notifications in case of any not standard behavior. There is also an OWASP guide covering this topic and of course OWASP AppSensor, which is a conceptual framework and methodology providing guidance to implement intrusion detection and automatic response into applications.

New Item: Insecure Deserialization

This is the second item included based on the community survey. While deserialization flaws are usually quite hard to detect and exploit, the impact can be devastating as it can lead to remote code execution, which is one of the worst attacks.

An application is vulnerable when accepting serialized objects (note that this does not apply only to java serialization but the process in general) from external sources. If a malicious serialized object is provided, it can lead to unexpected behavior such as remote code execution or complete system takeover. This does not apply only to inter-system communication, where serialization is involved, but also in situations such as caching.

The only robust protection is to use serialization solutions, where only primitive data types are allowed. If that is not possible, there are some ways to mitigate the risk such as

Logging all unexpected serialization inputs with automatic notifications to have early warning.
Deserialization modules should run with the least privileges possible.
Integrity checks of serialized objects to prevent data tampering

2013	2017
1. Injection	1. Injection
2. Broken Authentication and Session Management	2. Broken Authentication
3. Cross-Site Scripting	3. Sensitive Data Exposure
4. Insecure Direct Object References (Merged with 7)	4. XML External Entities (NEW)
5. Security Misconfiguration	5. Broken Access Control (Merged 4+7)
6. Sensitive Data Exposure	6. Security Misconfiguration
7. Missing Function Level Access Control (Merged with 4)	7. Cross-Site Scripting
8. Cross-Site Request Forgery (Removed)	8. Insecure Deserialization (NEW, Community)
9. Using Components with Known Vulnerabilities	9. Using Components with Known Vulnerabilities
10. Unvalidated Redirects and Forwards (Removed)	10. Insufficient Logging & Monitoring (NEW, Community)

More From OWASP

Okay, you are now familiar with OWASP Top Ten. You've read the document back and forth. You are now zealously making sure your apps are as safe as possible. What's next?

First of all, you should realize, that Top Ten is just a tip of the iceberg. You shouldn't definitely stop at 10. There are many, many more vulnerabilities and risks to look for. While top ten is good in raising awareness, it is by no means 'Security Bible'. For detailed guides and explanations, you'll have to look elsewhere. But where to start? With other OWASP projects, of course!

Some of the interesting projects are:

One more thing worth mentioning is that Top Ten is not suitable for a security verification checklist due to its limited scope. Turns out there is a better match -- an OWASP project specifically focused on this area - OWASP Application Security Verification Standard Project.

Documenting Angular apps with TypeDoc, CompoDoc and AngularDoc

Mon, 18 Dec 2017 22:12:03 GMT

There are several tools which you can use to document your Angular 2+ app. Let's compare them so you can decide what is the most suitable for your app.

Documentation in AngularJS

When working with plain JS applications, you would use documentation tools specially designed for JavaScript. One of the common ones is, for example, JSDoc. It is similar to JavaDoc tool used with Java, which can generate automatically documentation based on special comments, which are present directly in the code.

However, when working with a framework such as AngularJS (that is -- Angular 1.x), it is handy to have something more powerful. That means -- you need not only to document generic JS code, but you also want to take into consideration specifics of your framework.

For AngularJS, there is a special flavor of JSDoc called ngdoc. It adds a lot of angularJS specific stuff on top of regular JSDoc. It is then processed by a utility called Dgeni.

Documenting TypeScript with TypeDoc

Angular 2+ is way different than AngularJS. What's more, you usually write your code not in JS, but in TypeScript. That means you cannot use JSDoc nor ngdoc. You need a new tool. Turns out, that for TypeScript, there is a tool similar to JSDoc called TypeDoc. It is, however, angular-agnostic, so you cannot use any framework-specific goodies.

Getting started

To install TypeDoc locally for your current project, just run:

 npm install typedoc --save-dev

To be able to run typedoc as an NPM script, you need to update your package.json -- the scripts section:

"scripts":{
   ...
   "typedoc":"typedoc --mode modules -module commonjs --exclude **/*.spec.ts --out docs/typedoc src/app"
}

After this, you can run typescript generation for your current project by simply executing npm run typedoc.

TypeDoc tool will be executed in mode modules using common.js modules. Exclude defines a pattern which describes files to be omitted. It is useful for example for tests. Then using --out you are first defining output folder to which the documentation should be generated to (here it is docs/typedoc). The last is then the root directory containing the source code (here src/app). There are many more config parameters you can use (see the docs).

TypeDoc contains two themes out of the box. The default one and 'minimal'. To specify a theme use* --theme* option followed by the name of the predefined theme or a path to a custom one.

CompoDoc

While you can use TypeDoc to document Angular apps, it will treat your code as any other plain TypeScript app. That means no Angular specific documentation. But there is a whole lot to be documented. Compodoc solves this, it is focused on Angular apps specifically giving you a much more tailored solution.

Getting started

To install compodoc, simply run

 npm install --save-dev @compodoc/compodoc

Then update your package.json

"scripts": {
    "compodoc": "compodoc -p tsconfig.json"
}

Then you can run compodoc as a normal npm script using

npm run compodoc

Sample output

Once you run the compodoc command it will automatically generate your documentation from your project in form of HTML. Unlike TypeDoc, Compodoc understands that your app is an Angular application. That means it understands all the concepts such as components, pipes or directives. Bellow, you can see the generated documentation for a component.

You can see all the properties, constructors, and methods of the selected component. What's more, you can also view the component's source code, HTML template or resulting DOM structure. Since Angular applications consist of modules, it is vital to be able to document them and their relationships. Compodoc allows you to do that. You can even see relationships of each module in a nice diagram.

Compodoc also allows you to see a nice diagram of your application's routing, which is always handy.

One more feature worth mentioning is a documentation coverage, which allows you to measure how well commented your application is.

You can check sample Compodoc generated documentation - try the official live demo.

AngularDoc

There used to be another tool for documenting Angular applications called AngularDoc. It was similar to CompoDoc in a way that it would recognize both TypeScript and Angular-specific features. It could directly scan your Github repo and generate documentation available on AngularDoc site. At this time (Dec 2017) it is unfortunately unavailable as it is being reworked (the website still exists but you cannot register). According to the authors, we have something great to look forward to.

In the meantime, there are two similar products by the same company, which are still available and can be used. The first one is an extension of Visual Studio Code. The second one is called Angular Copilot. It is a standalone desktop app with similar features. Both of the products are good and helpful, but unfortunately, unlike TypeDoc and CompoDoc they cannot right away generate HTML documentation. Currently, they are more of a tool, which can help you during the development process on your local machine. For documentation purposes, we still have to wait for re-launch of the original AngularDoc. So let us stay tuned. This is a brief demo of AngularDoc VS Code Extension:

Conclusion

Having application documentation is very important and unfortunately often neglected. The problem often is, that it takes a lot of effort to keep the documentation up to date. If it is not zealously maintained, it tends to rot. Once it happens, the trust in the documentation is lost and it is next to useless. It is, therefore, vital to generate as much documentation as possible automatically with every change.

Since Angular applications are usually written in TypeScript, the regular JsDoc cannot be used. There is an option to use TypeDoc instead, which is a similar tool, but just for TypeScript. A much better alternative is, however, to use a tool which understands not only TypeScript but also the Angular Framework. Currently, the best tool for the job seems to be CompoDoc. AngularDoc used to be a viable alternative, but it is currently being reworked and not available. It should be worth it to keep an eye on it to see what the new version brings once it is finished.

Snyk – Detecting dependencies with known vulnerabilities

Wed, 22 Nov 2017 22:12:03 GMT

How to detect and fix security vulnerabilities in your dependencies using Snyk?

Introduction

Earlier, I published post Detecting dependencies with known vulnerabilities, where I was describing how important is to make sure your third-party dependencies contain no security vulnerabilities. And how to achieve it with OWASP Dependency-Check. This post describes another tool, which can be used to eliminate vulnerable dependencies -- Snyk.

Unlike OWASP Dependency-Check, Snyk started as a tool for managing NPM dependencies. It is actively developed and new features are often introduced. Currently (as of November 2017), Snyk supports not only NPM but also Java (Maven and Gradle), Ruby, Scala, Python and Go. However, the support for NPM is still the strongest, so many nice features are not available for other languages.

Unlike Dependency check, which uses National Vulnerability Database, Snyk has its own Vulnerability DB, which is also available on GitHub.

Similar to Dependency Check, you can easily integrate Snyk with your Build and CI tools, so you can break the build when new security vulnerabilities are introduced.

Installation and Setup

Before you can use Snyk, you need to install it using NPM. Make sure you have Node.js installed to do so. Then you can simply run:

npm install –g snyk

To run Snyk tests, you need to have a Snyk account and be authenticated. Just run:

snyk auth

This will open your browser's window, where you can create your account or sign in. Your security token will be downloaded and stored locally, so you don't have to auth the next time.

If you want to use Snyk on your CI server, you'll not be able to use this approach. In such case, you need to create your account on https://snyk.io and then you can go to My Account → General, where you can find your API token. Then just simply run Snyk auth with your token:

snyk auth [your token here]

Alternatively, you can store auth token directly in an environmental variable SNYK_TOKEN.

In case you want to scan your NPM dependencies, you also need to run npm install now before you can perform vulnerability tests. For Python dependencies, you'll need to run:

pip install -r requirements.txt

Running Test

Once you have your environment set up, you can run your tests. Let's start with:

snyk test

Snyk fetches the latest data from its own Vulnerability DB and checks all your dependencies for know vulnerabilities. The output may look something like this:

✗ Medium severity vulnerability found on org.apache.poi:poi@3.0-FINAL
- desc: XML Entity Expansion (XEE)
- info: https://snyk.io/vuln/SNYK-JAVA-ORGAPACHEPOI-31387
- from: com.vojtechruzicka:snyk-maven-example@1.0-SNAPSHOT > org.apache.poi:poi@3.0-FINAL
✗ Medium severity vulnerability found on xerces:xercesImpl@2.11.0
- desc: Denial of Service (DoS)
- info: https://snyk.io/vuln/SNYK-JAVA-XERCES-30183
- from: com.vojtechruzicka:snyk-maven-example@1.0-SNAPSHOT > xerces:xercesImpl@2.11.0
✗ High severity vulnerability found on xerces:xercesImpl@2.11.0
- desc: Denial of Service (DoS)
- info: https://snyk.io/vuln/SNYK-JAVA-XERCES-31497
- from: com.vojtechruzicka:snyk-maven-example@1.0-SNAPSHOT > xerces:xercesImpl@2.11.0
Tested 5 dependencies for known vulnerabilities, found 3 vulnerabilities, 3 vulnerable paths.

Monitoring for new vulnerabilities

When running snyk test, you are only notified about the currently known vulnerabilities, which are already part of the Snyk vulnerability DB. New vulnerabilities are, however, discovered all the time. Dependencies previously considered safe can contain high severity issues in the blink of an eye.

Snyk allows you to monitor your application's dependencies continuously for newly discovered issues. You need to run snyk monitor command:

snyk monitor

Snyk creates a snapshot of your dependencies and stores it. It then sends you an email whenever a new vulnerability, which affects you is discovered. You should run the monitor command every time you change your dependencies (e.g. after deploy job) to update the snapshot.

Addressing Issues

Update

If the vulnerability is not brand new, there is a good chance it was already fixed in a later version of your dependency. Updating dependency version is the easiest fix, but may not always be available. Snyk DB provides all the required information, so you can tell whether upgrading version will actually fix the vulnerability.

Patch

Sometimes, unfortunately, official fix in the form of a new version for the vulnerability is not available. Maybe the dependency is no longer actively developed or it is just too early after the discovery. Fortunately, nothing is lost and there is a good chance you will be able to still resolve your issue.

Snyk patches are one of the cool features, which makes Snyk better than other similar services. For some dependencies, the Snyk team actually provides their own fix in the form of patch -- a minimal set of changes required to fix the security vulnerability. This feature is unfortunately currently available only for NPM dependencies.

How does it work? The problem is that after applying the patch locally, the changes would be overwritten the next time you would run npm install. To prevent this, Snyk introduces a special .snyk file, which contains information about all the patches which need to be applied. Then after installing local dependencies by npm install, you just need to run snyk protect.

It downloads and applies all the patches contained in the .snyk file. So you need to run it after npm install but before the build. You don't need to edit the .snyk file manually, Snyk Wizard will do it for you (see below).

Ignore

You can decide to ignore a known vulnerability by running snyk ignore.

This may be handy in case there is currently no update or patch. Snyk will notify you via email when a fix is available later. It is a good idea always to provide a reason for ignoring (using --reason) and the duration short enough (using --expiry) so the issue is not buried permanently. By default ignore will expire after 30 days. Ignored issues are stored in the .snyk file.

Snyk Wizard

Running snyk test will find all the security vulnerabilities for you. It will not, however, fix them for you. Snyk Wizard comes to the rescue. You can run it with:

snyk wizard

This will go through all your vulnerable dependencies and offer:

Update the dependency to a newer version, which fixes the issue, if available
Apply a patch if available
Ignore
Skip for now

All your decisions will be applied -- package.json file will be changed to include the new dependencies and a .snyk file will be created with all the patches and ignores. Finally, the snyk monitor will be run to create a snapshot of your dependencies, so you will be notified in the future if a new vulnerability or fix is added. However, currently, only NPM dependencies can be handled by the snyk wizard.

Wizard's console output can look like this:

C:\projects\snyk-example>snyk wizard
Snyk's wizard will:
  * Enumerate your local dependencies and query Snyk's servers for vulnerabilities
  * Guide you through fixing found vulnerabilities
  * Create a .snyk policy file to guide snyk commands such as `test` and `protect`
  * Remember your dependencies to alert you when new vulnerabilities are disclosed
Querying vulnerabilities database...
Tested 1158 dependencies for known vulnerabilities, found 11 vulnerabilities, 30 vulnerable paths.

? 5 vulnerabilities introduced via aegir@12.0.0
- info: https://snyk.io/package/npm/aegir/12.0.0
  Remediation options (Use arrow keys)
> Upgrade to aegir@12.1.2 (triggers upgrade to debug@3.1.0, growl@1.10.3, tunnel-agent@0.6.0)
  Review issues separately
  Set to ignore for 30 days (updates policy)
  Skip

Integrations

While snyk Command Line Interface is pretty cool, snyk offers much more. It integrates with a wide range of external services.

Currently, snyk supports GitHub, GitHub Enterprise, GitLab and BitBucket Server as Version Control. From PaaS, there is Heroku, CloudFoundry, Pivotal Web Services and IBM Bluemix. There is even support for AWS Lambda (Google Could platform to be supported soon). Notifications about new vulnerabilities can be delivered via Slack or soon as JIRA tickets.

GitHub

Let's closely look how such integration works. We'll use Github as an example. After authenticating with your GitHub account, you will see your project in the Projects section of the Snyk webpage.

It shows all your dependencies and issues by severity. It will not only monitor your master branch, but it will also check all the pull requests to be sure that they do not introduce any new vulnerabilities.

What's even cooler, after testing your project Snyk will check its database whether there are any solutions for the issues found. That is -- if it can find any new versions fixing the vulnerabilities found. Or at least, if there are any patches. If there is something, what can be done, it creates automatically a Pull Request, which introduces the fix. How cool is that? You can see example PR here:

As with the Wizard, this is unfortunately only available for NPM projects though. Hopefully, other types will also be supported soon.

If your project is free of vulnerabilities, you can even include this nice badge:

Conclusion

Making sure your dependencies are up to date and contain no security vulnerabilities is very important and often overlooked. Snyk is a cool tool, which can help you with that. It is in some aspects similar to OWASP Dependency-Check described in the previous article. It has some nice features on top of what Dependency Check offers. Integration with a wide range of cloud services and version control systems. What's great is that it not only finds the vulnerabilities but directly offers means to fix them in the form of updates or patches. Hell, it can even automatically create a Pull Request for that. The downside is that many of the cool features are currently available just for node dependencies and not others like Java. We can only hope this will change soon.

IntelliJ IDEA Tips & Tricks: Postfix Code Completion

Tue, 07 Nov 2017 22:12:03 GMT

Postfix completion is a very powerful feature, which lets you transform an existing expression using a postfix added. This greatly reduces backward caret movement and thus increases productivity.

Postfix completion

Whenever you have an existing expression, you can append a specific postfix after a dot and apply it using a Tab key. IntelliJ IDEA takes the expression and transforms it based on the postfix provided.

Let's say you have a list of strings called items. You can then type items.for *and then press Tab. It creates a *for-each iterating over the items.

Pretty cool, right? Note that the resulting for-each has the proper type of items - that is String. Or let's have another example - *.try *postfix. It surrounds an expression with a try-catch block.

There are many more templates available, such as (for Java):

fori - for iterating with an index over a collection
forr - iterates with index in reverse order
switch - produces switch statement with the expression
sout - Prints the expression using System.out.println
null - check if an expression is null
notnull - checks if an expression is not null

It looks cool and all but what is it good for you say? You can achieve similar results using the regular live templates in most cases anyway. Often you realize later, that you need to do something with the expression you have already written (maybe add not null check just to be sure?) and it saves you backward caret movement. Also, for many people, it is more natural to think in order of - subject and THEN action which needs to be performed with it. In a given context, there are usually very few subjects to choose from (keep your methods short, remember?), but many actions available. It is easier, when you are offered just the relevant actions when triggering postfix completion.

The feature was originally available in ReSharper, then as a separate plugin for IDEA and was later introduced as a core part of the IDE in version 13.1. It is currently (as of 2017.2.5) available for Java, Scala, JavaScript and Kotlin.

Displaying available templates

It is not necessary to memorize all the postfix templates. Instead, you can use a handy shortcut to show the available postfixes: Ctrl + J (or ⌘ + J on Mac). It will open a popup, which only shows items relevant to the given context, not all the existing templates. That means - if you open it when the cursor is next to a collection type, it will only show postfixes applicable to a collection.

Configuration

You can browse available postfix templates in Settings under Editor → General → Postfix Completion. You can disable individual templates or the feature as a whole.

Writing Custom postfix templates

While IDEA offers quite a lot of built-in templates, it does not, unfortunately, allow you to define your own. But fear not, there is an IDE plugin just for that. It's called Custom Postfix Templates.

The plugin comes with a wide variety of additional templates, which you can use or edit to better suit your needs. But what's more - you can actually add your own. The plugin currently supports Java language, but according to the roadmap, more are to follow.

What's really nice about the plugin is that the template syntax is really easy to read and write. Let's look at a specific example:

.toInt : convert to int
    java.lang.String → Integer.parseInt($expr$)
    java.lang.Number → $expr$.intValue()
    NUMBER → ((int) ($expr$))

The first line consists of postfix, which triggers the template (.toInt) and after the colon, there is a text description. Then there is a number of lines, which define what result should be generated based on a different input type to which is the postfix applied.

That means that the same postfix can define different results when applied to different types. In the example above, you can see that converting String to Integer using .toInt postfix will result in Integer.parseInt($expr$). However, applying the same prefix to a number will result in a type cast ((int) ($expr$)).

The plugin adds a new section in Settings under Editor → Custom Postfix Templates. There you can see the list of all the templates and also edit it in case you want to alter them or add your own. What's handy is that the plugin will show you a DIFF against the default state, so you can tell what you've changed.

Conclusion

Postfix code completion is a really powerful feature, which can boost your productivity quite a bit. Remember, the list of all the available actions in the current context is available via Ctrl + J (or ⌘ + J on Mac). And if the built-in actions are not enough for you, be sure to try Custom Postfix Templates for IntelliJ IDEA plugin.