Jazoon 2013 – The Economies of Scaling Software

25. October, 2013

Jazoon 2013 badgeIf you’re small, scalability is not an issue. If you’re big, you can plan for it. But what if you wake up one morning and you suddenly find you’ve become the next Google?

In his talk “The Economies of Scaling Software“,¬†Abdelmonaim Remani talks about what scaling means, how you can plan for it (even if you don’t expect it to happen) and all the nasty details that stand between you and success (slides on slideshare).

Today, the ubiquity of the Internet has blurred the lines between consumers and enterprises. They start to ask the same questions which eventually all boil down to: “How can I find what I’m looking for in a universe of haystacks?”

One day, many of us will find themselves with the need to scale because the performance of the old solution has become unbearable and all low-hanging fruit (faster CPU, more RAM) have been picked.

To solve this, you can look at the CPU (slide 16-35) or you can start to build clusters (36-45).

Or your I/O might be the bottleneck (slides 46-60). You can solve this by looking at NoSQL databases and caching.

Is the network the problem? (slides 61-75) Start with asynchronous processing, batch jobs, content delivery networks (CDN), DNS sharding or use a different protocol to connect the various parts of your system.

But how do you know what part is the bottleneck? The answer here is monitoring. (slide 77)

Note that scaling often helps with disaster recovery, you still have to plan for it – if all nodes of your cluster are in the same room, it’s still a single point of failure (slide 79)

Software isn’t everything. Don’t forget your team (slide 82)

Jazoon 2013 – Join the Java Evolution; JCP & Adopt-a-JSR

25. October, 2013

Jazoon 2013 badgeHeather VanCura explained how the new JCP works in her talk “Join the Java Evolution; JCP & Adopt-a-JSR” (slides on slideshare)

Oracle spent the last years to make the JCP much more open and accessible. One example here are the JSRs which are much more lightweight than the complex JCPs. You can even adopt one.

Maybe you’re a long time joda-time user and want to make sure important features make it into JSR 310 – new Date and Time API for Java? Join them to discuss your need, share some code, help write, improve or translate documentation.

Some bug in the SDK nagging you? Contribute a fix to the OpenJDK. With Java 8, the OpenJDK build has been simplified tremendously. Sun ignored your bug report for years even though it contained a patch? Now is the time to change this.

I had the chance to chat with Heather after the talk which earned me a copy of “Java 7 Concurrency Cookbook”; thanks for that ūüôā We discusses a couple of ideas and she gave me points; maybe I’ll submit a few patches. If I do, I’ll blog about the experience here.

Jazoon 2013 – Real World Git Workflow

25. October, 2013

Jazoon 2013 badgeWhen using git, you really have to define a workflow that you want to use. In his talk “Real World Git Workflow” (slides on slideshare),¬†Stefan Saasen explains some kinds of workflows and when they are appropriate.

The first step in any workflow design is asking questions. Some examples that will sound familiar:

  • “Can we fix a bug for a specific release?”
  • “Can we do a fast hotfix for the current release?”
  • “Can we build the current code?”
  • “Is the code for that feature complete?”
  • “Has everybody reviewed the code for this feature?”

Your questions will be different but make sure you ask them. There is no perfect workflow but there is your workflow. Omit this step at your own peril.

Collaboration Models

First, which collaboration model do you use?

“Anarchy” (anyone can push anywhere, slide 17), “Gatekeeper” (one person reviews all changes, often used by OSS projects, slide 18), “Dictator and Lieutenants” (Linux, slide 19) or “Centralised” (slide 20). See also “Git Workflows” on atlassian.com/git.

In the enterprise, the centralized model is often used. This approach makes it most simple to integrate all the tools (CI servers, code quality tools, deployment, …). (slide 23)

Branching Models

The two most common branching models are “continuous delivery” and “product releases.” (slide 25)

From slide 28:

Significant branches¬†map to a concept in the outside world. It may be a past release, an environment or a role. Those branches are long-running and stable whereas feature branches are short lived and volatile.”
– Stefan Saasen

Slide 27 shows the branches for¬†“continuous delivery”. PR is a “pull request.”

As you can see, development is consolidated in the staging branch and pushed into production (master branch) from there. If you make a hotfix in the master branch, it is cherry-picked back into staging.

Slide 30 shows how to handle¬†“product releases.” You have a single, central repository with a master branch which consolidates the development (no staging). Each feature and bugfix happens in a short-lived “feature branch.” When a release is made (slide 31), then a new long living release branch is created. Bug fixes still happen in short lived branches.

When the bug is fixed, the fix is first merged back into the oldest affected release. From there in the next release until you get to the latest release from which it is then merged back into master (slide 33).

But what about changes you don’t want to merge like the changes made by the Maven release plugin? Maybe you will want to replace it with a better release workflow.

Or we use the correct git merge strategy: ours. This creates a new changeset with the merge information without actually merging anything. For git, it will look as if everything has been done and it won’t bother us with merging those changes ever again (slide 39).

Merge Protocols

While the above sounds reasonable, the question is why? What are the rules and forces which make this better? Stefan introduces “the merge protocol” to answer this. See slide 42 for this.

In a nutshell, you always try to merge more stable branches into less stable ones: Bug fix branch into branches where the bug hasn’t been fixed, yet. Features into branches which don’t have the feature.

That’s why you never merge master back into a release branch: Releases are most stable. You merge them into master. If you have fixed in master that you really need in a release, you cherry-pick them (slide 43).

Pull Requests

A lot of people understand why code reviews would be a good thing, “but …” Sounds familiar? Then pull requests are for you.

Pull requests are an easy, low-overhead tool to have as much “code review” as you feel comfortable with. You can merge with by clicking a button or you can review the changes line-by-line. Your choice.


Most projects will use a single canonical repository but remote forks are useful, too. Imagine you have fixed an important (for you at least) bug in a OSS project. You send them a pull request but it’s rejected! What do you do?

You fork the project. git allows you to still track the changes made by the original project while isolating your life as much as you want (slide 52)

A fork is nice if you want to do an innovation spike – code that might never be included in the product. Fork instead of polluting the project history with dead experiments (53).

Some department needs big changes to some component? Fork it until the feature stabilizes. You can¬†still merge them if you want, but you don’t have to (54).

Reduce the noise (55). A fork allows you to rewrite history.


You can use pre and post hooks to make everyone’s life easier. Use a local pre-commit or pre-push hooks to make sure some important tests have been run. For example, you could run FindBugs or checkstyle.

An interesting post-checkout hook would be to check whether the branch is green (66), i.e. code builds and all tests pass. Stop wasting time to search for bugs that were already there before you started your work. You can get this gem from bitly.com/green-builds (69).

Continuous Integration

The explosion of branches can quickly bog down your build server if you don’t come up with a strategy to handle this (71). Usually, it’s enough to build stable and master but developers will love it when they can manually trigger feature branch builds (72).

Jazoon 2013 – Kafka and Storm – event processing in realtime

25. October, 2013

Jazoon 2013 badgeGuido Schmutz introduces two frameworks in his talk “Kafka and Storm – event processing in realtime” (slides on slideshare)

Apache Kafka is a publish-subscribe messaging system like JMS but more simple. Messages are kept in files and never deleted. Together with the fact that subscribers have to tell the system which message they want next, this means you can recover from bugs that corrupted your data even if you notice them only after some time: Just process all the corrupted messages again.

Storm is a “distributed realtime computation system.¬†” It makes it easy to define topologies (= graphs) of bolts (= places where a computation takes place) flowing your real-time data through a complex network to process, filter and aggregate it. Just like Akka, it defines all kinds of operations (filters, switches, routers, …) so you can easily and quickly build the topology you need. Trident makes this set-up step even more simple.

Compared to Hadoop, Storm is meant for real-time processing. Some projects combine the two.

If you need a good framework for serializing data in Java, have a look at Apache Avro.

Jazoon 2013 – Rocket Propelled Java

25. October, 2013

Jazoon 2013 badgeRocket Propelled Java” is about being more productive by writing less boilerplate code (slides on slideshare).

In his talk, Andres Almiray showed some tools to make your code more compact while keeping or even improving readability.

The first one is lombok and lombok-pg which use Java annotations to generate a lot of boiler plate code (slides 3-6)

Similar, you can use Groovy AST transformations to create things like immutables or singletons correctly (“There are 5-6 ways to create singletons. Only two of them are correct”,¬†slides 7-9)

Note: You might be worried to add Groovy to a pure Java project. You don’t have to. Just use it for the AST transformations to stop wasting time on writing code that only makes your compiler happy. Use Groovy as a “boilerplate code buster.”

If you ever wanted to use “Design by Contract“, GContracts is for you.

Jazoon 2013 – 33 things you want to do better

25. October, 2013

Jazoon 2013 badgeTom Bujok listed a lot of methods, technologies and frameworks that you should be aware about in his talk “33 things you want to do better” (slides on speakerdeck)

At the beginning he reminded us how quickly a well designed system goes bad due to hurried changes. We need to be aware of our technical debt and we need to allocate time to spend on reducing it (slides 3-12).

As an example, car batteries are easy to find. They are a replacement part, designers and engineers make it easy to find. Compare this to the configuration of your project. If you need to change it, how easy is it to find the file that needs to be changed and then the place in the file?

Another important point is skills. In most other professions, you have some mastery of a skill before you use it. You train hundreds of hours before you play your first football game. In Software, we show you a computer, we show you the programming language of the year (not necessarily¬†this year’s). There is no time to master the tools you have to use from day one (slides 13-15).

“We are what we repeatedly do; excellence then, is not an act but a habit.” – Aristotle

Or as Wikipedia defines it:

“Habits […] are routines of behavior that are repeated regularly and tend to occur subconsciously.”

Stop wondering why you always make the same mistakes – they’re habits. Eliminate them ASAP (slide 19):

Bad Habits – Katherine Murdock “The Psychology of Habit”:

  • Recognize bad habits and eliminate them ASAP
  • The older you get the more difficult it is to remove a bad habit
  • Each repetition leaves its mark!

Turning bad habits into good ones – Dr. Michael Roussell, PhD.:

  • You can’t erase a habit, you can only overwrite one.
  • Insert the new habits into the current habit loops

Bad Habits

Configure your IDE properly and remove bad defaults. Replace “ex.printStackTrace();” with “throw new RuntimeException(ex.getMessage(), ex);” (slides 43-45).

One bad habit is empty catch blocks with “can never happen” comments. If you see one during a code review, replace it with “System.exit(-1);”. It can never happen, right? Right? (slides 46-47).

Note: I have create a “ShouldNotHappenException” for this case ūüôā

Another one is to make every method in a static helper class public. Maybe some of them can be package private? (slide 48)

Learn about other good habits. Read books like “Effective Java” (Joshua Bloch) and “Clean Code” (Robert C. Martin) (slide 49)

Use code reviews to notice bad habits and to spread knowledge in your team but prevent blame games. (slide 73)

Learn the keyboard shortcuts of your IDE (slide 78)

Remember (slide 79):

“Any jackass can kick down a barn, but it takes a good carpenter to build one.” – Sam Ryburn

Projects You Should Know

Project lombok and lombok-pg РIn a nutshell, these hook into the Java compiler and generate additional bytecode when certain annotations are present. Bored with getters, setters, hashCode() and equals() plus a nice toString()? Use @Data (slides 21-28).

Guava (slides 29-33)) is a great library with many tools that you have been missing in Java for years. You might also want to look at commons-lang.

Want to use lambda expressions but can’t upgrade to Java 8? Then lambdaj¬†is for you (slides 34-39).

Logging? slf4j (40-42). Especially nice when combined with the @Slf4j annotation from lombok.

Bored to write all that boiler plate code to create all those services and managers that form your app? Look at Guice or Spring.

Use Spock to make tests more compact and easier to understand. (slides 50-52)

Unitils contains all the helper functions that we always missed in JUnit and Hamcrest (53-56).

JUnitParams will help you run tests with different parameters (57-59).

Need to wait for something during a test? Awaitility will help. (60-61)

When mocking isn’t enough and you need to inject code during a test, Byteman is the tool you want to look at (62-63)

Getting bored writing boiler plate code in Java to make a compiler happy? Have a look at Groovy. (64-67)

How about adding dependencies to your scripts? Try Grape. (68-69)

Is your build a mess? Do you feel Maven is too verbose or too limiting? Gradle might be for you. (70-72)

Version control is slowing you down? Have a look at Mercurial or Git (75)

Use bash or Python to automate man-/menial work. If you’re on Windows, look at Babun or Cygwin. (76-77).

Jazoon 2013 – Open Safety For Drones

25. October, 2013

Jazoon 2013 badgeDrones – fully or partially autonomous aircraft – are the second most wide-spread form of private-owned robots today only challenged by room-cleaners. The current estimation is that there are 500’000 drones currently in use. Parrot AR alone sold about 250’000 units. In his talk “Open Safety For Drones,”¬†Lorenz Meier asks important questions about safety, regulation and how this area will develop in the future.

When it comes to safety, then there are two areas: Safety when developing drones and safety when using them. People have started to demonstrate the usual stupidity by flying drones in crowded areas without regard to safety. Lorenz showed a video where an inexperienced controller flew a drone in Manhattan until it crashed into skyscraper and dropped into the streets below. In another example, someone flew a drone through a campus. I say “through” because he went below footbridges that connect¬†the first floor of buildings, eventually passing a bus on the way. The “pilot” also failed to catch the drone after the stunt, crashing it into a hedge.

It will be interesting to see how legal bodies like the FAA respond to things like this.

Despite this, autonomous drones are already useful. A nice example is the new 3D model of the Matterhorn.

A more related issue is the other one: Building safe aircraft. Bugs in a drone can really ruin your day (or life). How can we make sure that hundreds of thousands of lines of code have an acceptable number of bugs?

The issues here are many:

  • If you want to fly an autonomous aircraft, you need to license it.
  • Regulation for drones are non-existing or immature. They might try to handle you like a normal aircraft manufacturer (think Boeing or Airbus). Can you even afford to apply for such a license?
  • You’re using an image recognition software. It uses a math library. In that library, a single line of code has changed. What is the impact on the safety of your drone? How do we bring safety and the need for faster iterations together?
  • Regulators might insist that you license the whole thing again.
  • To get real engineers on your project for a new drone, your API must be really simple. As every seasoned developer knows, nothing is harder than “simple.”

A word on the safety of military drones: Not a single one of them is allowed in civilian airspace. But it’s probably very expensive. Germany spent ‚ā¨562 million to certify the EuroHawk drone. The project was cancelled and the defense minister “referred to the project as being ‘a horror without end'”.

So how can we make a DIY drone ever safe?

There are some approaches:

  1. Involve as many people as possible in a project. Everyone will find something that can be improved.
  2. Allow commercial use of your project. Companies have completely different views on safety than hobbyists.
  3. Collect logs of your flights and share them. Share your project on github and make sure you include the hash of the changeset that was used to run the drone.
    Over time, you can use statistical data to locate even “once in a million years” bugs.

But there are good news, too. With a plane, you have X people on the aircraft and 0 on the ground (most crash in empty areas). With a drone, you have 0 people on the aircraft and X on the ground. But using the laws of physics, it’s pretty simple to estimate the maximum radius that a drone can still travel if it’s “brain” shuts down right now. That means we can test them pretty safely in a virtual cage. Also, since all the components are open, we can run extensive simulations before bringing the thing into the wild.

But the OSS approach also has drawbacks. With OSS, a unit test is easy to write, the test case is known and if something goes horribly wrong, we might experience data loss.

With unmanned systems, the worst case is loss of life. And where do you get your test cases from? How do you plan to test for a broken coil in an actuator? An overheated battery which results in unstable power? Wind gusts? Collisions with birds? Interference with other electrical systems?

Maybe it’s time to open a new field:

Open Safety – Build a safety track record in confined test areas in an open, community-driven process, and only let statistically proven safe systems out of the sandbox. SHARE validation results on the library and version level across systems.

Open Safety Manifesto

  1. Dropping is Free – Ground impact must only damage vehicle
  2. Anytime Shutdown – A separate shutdown unit is present to enforce a drop
  3. Built to Drop – Battery protected against impact, parachutes, etc.
  4. Every Flight is Logged
  5. Every Log is Reviewed – Can be done on Droneshare

State of the Art

Sites like Droneshare have collected so many flight hours for some drones to satisfy normal regulation rules.

Keep in mind that certification alone doesn’t make things safer, they just help to catch the most common errors. Simulations and test protocols improve the situation further. Lastly, lots of people mean that every part is tested in thousands of ways. Software developers know how quickly someone else finds problems with their work (one reason why pairing is so unpopular).

A big problem is YouTube since it teases out the “I more stupid” behavior in humans.

Military (Ab)Use

Another issue is the (ab-)use of this work in military drones used to kill. The scene is aware of this but they currently feel that the good (like cheap access to aerial photography, autonomous mesh networks built by drones in disaster areas) to outweigh the risk. A more visible discussion about OSS and hardware in military projects would be welcome. Some projects forbid military use (Gnutella). It’s worth noting that the US Navy has started to buy Linux based drones. Personally, I find things like the Samsung SGR-A1 military robot sentry (with 5.56 mm robotic machine gun and a grenade launcher) disturbing. Especially when it’s being advertised to be able to¬†“identify and shoot a target automatically from over two miles (3.2 km) away.

Jazoon 2013 – How To Do Kick-Ass Software Development

24. October, 2013

Jazoon 2013 badgeOne of the most important aspects of agile is to notice your problems and solve them. How? A good strategy is to look what others did. In “How To Do Kick-Ass Software Development,” Sven Peters shares his experience with “Kick-Ass Software Development at Atlassian” (slides on slideshare, video on vimeo). Yes, he have the talk wearing the “Kick-Ass Super Hero Costume“.

A short refresher what agile aims to be (slide 12):

  • better software
  • less overhead
  • faster development
  • happy customer
  • happy developers

This only works when you continuously improve. Legacy teams, can you hear me?

Kick-Ass Software

But before we can start: How do we know we build the right thing? Bad example: The Microsoft Kin. Never heard of it? They sunk about $1 billion in the project. Makes you wonder how you could miss it, right? (slide 25)

Better solution: Fake it. IBM made extensive experiments with a state-of-the-art speech-to-text system before starting development. How did they do it? They hired an extremely fast typist who remote-controlled word. Cost? Negligible. Result? Text-to-speech won’t sell – people can’t edit the text fast enough using words, they are hoarse after eight hours and the whole thing really breaks down in an open office space. (slide 27)

Use paper prototypes. Something geeks eventually forget: You can drag a piece of paper over a (wooden) desktop just like a window over a screen. (slide 28)

Quick and easy feedback: Easy to find, simple, fast to submit (slide 32). Customers love it! But remember to protect your developers. Gmail uses the “Sh*t umbrella” (slide 38). But not everyone is Google. Put each developer into a support seat once in a while. Not only can they solve problems much more quickly, they also learn to look at their work from the customer perspective (slide 42).

Kick-Ass Team

In most companies, there are way more developers than testers. Atlassian has a ratio of about 13:1. This creates bottlenecks and issues with accountability and scalability (slide 46).

Solutions: Dev X tests features of other devs. (slide 49)

Use “DoTing” – Developers on Test (slides 50 -56):

  • Give developers the same training QA receives
  • Pair testers with devs
  • “Blitz Test” – Once in a while, invite everyone in the company to try the product
  • Use test recipes
  • Split sessions (1 dev and 1 QA test the same feature independently)
  • Bug Hunter (have someone try really hard to break the app)

Remember “Quality is everybody’s responsibility.” (slide 57)

At the Jazoon, the slides about design were skipped due to time constraints.

Again: Department barriers slow you down. Work together as one team! Keep improving! (slide 69, see also: WIKISPEED in “Test First Saves The World”)

Kick-Ass Collaboration

The lonesome cowboy coder is a relic but teams always mean trouble. Interesting video of a heavy-traffic crossing without signals on slide 73. “Traffic Rules protect us from accidents” -> “Development Rules are protecting us from making mistakes”. Like we need a concurrency library in our preferred language, we need a “Fast + Simple Workflow For Parallel Coding.” Otherwise we will be slow and some people will enable their “god mode.” (slide 77)

DVCS make this simple. Create a branch for every task (slide 78). They should have short lives (2 days).

“Pull Requests can improve your code quality” (slide 80). This helps to spread knowledge. But keep an eye on the blame game!

Everyone in the same room would be great but it’s not always possible: People travel. You really want to pull some experts two weeks out of his family? (slide 85) Ask yourself: “Where do I work best?” The answer often is: It depends¬†(slide 86). Make sure people can stay in the zone when they have to.

Make sure you can communicate effectively (slide 87). Emails look great (slide 88) but they cause problems (slide 90). Try chat. Invent your own Portal Device.

Most importantly: Remove friction!

Kick-Ass Automation

Developers automate everything for everyone else. Spend some time each week to automate a menial/manual task for them (slide 101).

4 steps to tame monster builds (slide 104-110):

  1. Instead of having everyone build everything, pass the artifact
  2. Run your tests in parallel. They are independent of each other as they should be, right?
  3. Have a build strategy. How often do you build what part?
  4. Make your stats visible. Use wallboards, information rediators.

Automatically disable flacky tests (113). Use SonarQube and similar tools to detect problematic code. Use Freud to find your own anti-patterns (115).


Remember managers are also humans (123).

Share stories of success and failure (124).

Remember: To be awesome, you have to step out of your comfort zone (126)

Jazoon 2013 – User Storytelling: The Lost Art of User Stories

24. October, 2013

Jazoon 2013 badgeIn her talk “User Storytelling: The Lost Art of User Stories“,¬†Ulrika Park (blog, slides on slideshare) made a case to become aware of the spreading practice of “snapshot user stories.” As an example compare “give warrant to use the bonus” (slide 14) ¬†to slide 16:

Elisabeth sees in the MyStore newsletter that she this month has got 12‚ā¨ in bonus and that she has a total of 16,7‚ā¨ in bonus on her club card.
E’s husband Claes go to the store. He has a club card too. “You can probably pay with your card” says E.

Claes shops. When he will pays, no bonus is withdrawn from the sum at the cashier.
So he pays the whole sum and gets home. “Strange why can’t I pay with the bonus, how does this work”

E already has a registered user on mystore.se Sometimes she logs in to see her credit card balance.

E logs in and see the saldo “16,7‚ā¨”. She clicks the balance and enters “events page”

There she sees that she’s bonus owner and that her co-bonuxcollector is Claes.

She gets information that she needs to sign warrant for him and choose the option to print a form.

She choose to print the form, checks Claes, signs the form and goes to the mailbox the day after.

2 weeks later she gets a letter from MyStore that confirms the warrant is verified. A week later Claes goes shopping. This time, the bonus is withdrawn from the total buy.

I think even without the highlighted parts, you can quickly see that the whole story is much richer than the “more efficient snapshots.” Moreover, it helped to expose some critical problems that no one noticed before:

  • Hidden expectations (user must already be registered)
  • This process takes four weeks! Do we really want that?
  • It helps more people. QA will be able to derive test cases from this. Developers will know much better what is expected from them. BAs will know which questions to ask.
  • It’s pretty easy to follow it, even if you’re not a software developer. Remember that these stories were meant as a means to communicate better with customer and managers?

A good story has these properties:

  • A hero
  • A platform
  • An enemy or challenge
  • Emotions (“how does this work?”)
  • Allies (cashier, web site)
  • A mission to accomplish

Everything is held together by a logical sequence of events.

Related articles:

Jazoon 2013 – Big Data beyond Hadoop – How to integrate ALL your Data

24. October, 2013

Jazoon 2013 badgeBig Data is being used everywhere. Kai W√§hner mentioned a couple of examples in his talk “Big Data beyond Hadoop – How to integrate ALL your Data” (slides on slideshare):

Anyone else getting worried by these “success stories”? How do you feel as a mobile customer that your mobile company tries to prevent you from leaving? How about using Big Data to notice bad customer service and prevent making customers unhappy? How do Macy’s competitors feel about this “monitoring”?

Anyway …

One great point was “Silence the HiPPOs” (highest-paid person’s opinion). With the ability “to interpret unimaginable large data stream, the gut feeling is no longer justified!”

Why Big Data? 3 V’s: Volume, Velocity, Variety. But don’t forget the fourth: Value (slide 8)

Before you can start analysis, you need to get the data from somewhere. That usually means integration of a foreign system (reading the data), manipulation of the data (like string to int or date conversion, etc.) and filtering (duplicates, importance, …). See slide 9.

Beware that Big Data is no silver bullet. If you have a gigantic amount of data with poor quality, that will just give you huge problems.

When planning for a Big Data project, begin with a business opportunity (slide 22). Chose the right data (don’t just import everything because you might need it), combine different sources and use easy tooling (slide 26).

Be wary of ETL tools. The network will quickly become your bottleneck.

For the actual implementation, he suggested to look at Apache Camel (slide 34) as a pure integration framework and the talend Open Studio (slide 56) as an example of an integration suite.

%d bloggers like this: