What if I told you, one could create and start using an Oracle Database, all from their browser, with NOTHING to install?
Now I know what you’re going to think, oh, he’s going to pitch SQL Developer Web again. And yes, I could, but instead I want to talk command line interfaces, specifically, SQLcl.
Zelda is the BEST.
CLI’s are less heavy than a GUI, generally, quicker, and simpler. Plus you’re a nerd, and you all know that you get max nerd HP+ when you work from a command prompt!
When you’ve logged into the Oracle Cloud (OCI), you get an immediate shortcut to open a shell prompt directly in your browser – this is called Cloud Shell.
Check the upper right hand corner of your OCI console.
Click that shell prompt looking button, and… Boo-yah!
Now remember, I said ‘NOTHING’ to install. So how do we get into the database (this would be a database that you have already created) ?
Our modern Oracle Database CLI is already installed and ready to go in Cloud Shell, simply type ‘sql’.
Now, before we do that, we need to get our Database Wallet. For maximum security, this is required to access your Autonomous Database – inside or outside the OCI network.
If you know your database’s ID, then you can easily ask OCI to download your wallet file directly to your Cloud Shell environment – just use the ‘oci’ command. (Docs)
Here’s an example, and no, that’s not the id for my ATP database, nor is my luggage combination 1-2-3-4-5.
oci db autonomous-database generate-wallet --generate-type ALL --file mywallet.zip --password CaN3s0v3rBrU1nS --autonomous-database-id ocid1.autonomousdatabase.oc1.iad.abcdefghijklmnopqrstuvwxyz123
And with that, the Wallet archive is in my CWD!
Inside that Zip file are a few things – the actual wallet required for a SSL (encrypted end to end conversation with your database) and a TNSNames.ora file are the most interesting things for getting connected to our database.
SQLcl is simply launched with the ‘sql’ command.
We want to start it with the /nolog flag, because first we’re going to setup our ENV for the connection. That’s accomplished, easy-peasy with
set cloudconfig mywallet.zip
Our TNS info is immediately available. We can see what databases are defined by using the SHOW command.
show tns
What comes back is a list of the services for our autonomous databases. My Autonomous Transaction Processing instance has a TP Urgent, TP, high, medium, and low service. Each of these have different affects on how your session will be allocated system resources – things like parallel processes for executing SQL and CPU.
The database expects a password, it prompts for it, and it I get it right, I’m in!
If this is your first time into your database, use the ADMIN account. I’ve been in before, and had created a database user called ‘JEFF,’ so i’m using that.
Session Times Out? That’s ok!
Better than ‘ok’, it’s good! If you’re away from your browser/machine, you don’t want to leave your system vulnerable – even if it’s just your pets or curious kids.
If your session does time out, you can reconnect, and your ZIP fill will still be there.
I’m lazy, so I like this.
Now what?
Now you can start working with your Oracle Database.
create users
create tables, stored procedures, views
load data
run queries
files files as blobs to your table(s)
export tables as local json, csv, xml files
see what you have/poke around
Let’s try a few things, right quick.
Firstly, your SQLcl command history WILL persist across sessions. This is handy, because remember, I’m lazy.
You can simply type ‘history’ to get a list of everything, or you can Up or Dn Arrow through the SQL history right at the prompt.
And while I’m in here, I’m going to setup my CLS alias, because typing ‘clear screen’ gets old.
‘history 3’ would put the CTAS in the buffer for me to execute, edit, etc.
If you have a habit of doing CONNECT in your SQL sessions, no worries, SQLcl is smart enough to NOT write those to the history, so your passwords will remain off the filesystem.
Now let’s create a table, put some data in it, and spool it out as JSON to the CWD.
json-formatted is somewhat ‘new’…easier to read for us humans.
Exit from SQLcl, and check out our new json file.
Remember, I could have chosen json, csv, xml, insert statements, html, or even a custom format.
Let’s go back into our db and see just what that DD table is.
info+ would show us column stats vs comments.‘tables’ is an alias for ‘select table_name from user_tables’
As you can see, the Autonomous DB collected stats for me on my table and had a fun time creating histograms for quite a few of the columns.
Don’t have a database yet? You can create that from the CLI too.
My colleague Todd has a great post that shows some of this, but starts by actually creating a new Autonomous Database instance from Cloud Shell too!
Todd’s a great follow for learning any/everything on OCI if you’re a developer. He’s a polyglot, so expect a bit of everything in his feed. Or follow him on Twitter, https://twitter.com/recursivecodes
Not only is it still a serious problem, it’s a major component of the NUMBER ONE problem according to the fine folks at the Open Web Application Security Project (OWASP).
We at Oracle have been talking about the perils of SQL Injection in your code, specifically PL/SQL since that’s OUR programming language of the Oracle Database, for more than a decade.
As a quick aside, I’m not joking about the ‘more than a decade’ and ‘bind variables’ – Tom first answered that question in 2004, and almost every single talk he delivered to customers featured the benefit of Bind Variables!
Seriously though, if you’re curious about how to make your PL/SQL code more secure, then we have this White Paper that tackles the subject in greater detail.
Tools can help!
SQL Developer, your PL/SQL IDE, will show you when things are less than good. And we adopted this feature back in 2018.
As George Takei would say, Oh My!
In SQL Developer, our parser is trying to highlight the issues before you even have a chance to compile the code – note the ‘grammar squiggle line’ underneath the block1 in line 18.
SET CODESCAN in SQLcl
SQL Developer might be your PL/SQL IDE, but SQLcl is your modern command line interface for the database. And I imagine you might be deploying a ton of PL/SQL through it to your database.
You’ll never guess what THIS does
Here’s an example of some vulnerable PL/SQL code straight from the Oracle Docs –
CREATE OR REPLACE PROCEDURE get_record ( user_name IN VARCHAR2, service_type IN VARCHAR2, rec OUT VARCHAR2 ) IS query VARCHAR2(4000); BEGIN -- Following SELECT statement is vulnerable to modification -- because it uses concatenation to build WHERE clause. query := 'SELECT value FROM secret_records WHERE user_name=''' || user_name || ''' AND service_type=''' || service_type || ''''; DBMS_OUTPUT.PUT_LINE('Query: ' || query); EXECUTE IMMEDIATE query INTO rec ; DBMS_OUTPUT.PUT_LINE('Rec: ' || rec ); END; /
Anytime you see something being appended into a string which is then later executed…that’s a big, red flag.
Now, if we send that through to the database from our SQL prompt, we’ll see this –
I like how we ‘connect the dots’ for you. Lines 2, 11, and 17 could allow very bad things to happen!
Code Reviews are expensive, but worth every penny!
If you aren’t able to spend as much time as you’d like reviewing your application source code by hand, at LEAST deploy tools that can do automated scans. There are plenty of them out there.
Just because you’ve been hearing about SQL Injection for 15 years doesn’t mean that it’s not a problem any more. Perhaps the best thing you can do is help your junior staff out by showing them the patterns to avoid in their code.
SQLcl is a Java application. The Oracle Java Virtual Machine (JVM) and JDBC driver looks at your Operating System to determine things like what language should be used for your user interface and your NLS_LANG setting for your Oracle session.
But.
Many of you like to keep your OS in your native language, but reserve your development tools to English.
So, without forcing your email, browser ,and everything else on your machine to a different language, how do I tell SQLcl to use the language I want?
Then, even if your OS was setup for Simplified Chinese or Portuguese, your SQL Developer user interface would be in English.
However, there is no sqlcl.conf – we have a compiled binary you can run.
So, if you want to set any JVM flag for your instance of SQLcl, you can set this Environment Variable.
JAVA_TOOL_OPTIONS
SQLcl in Spanish
There are hundreds of languages supported in general, but for our database tools, we support 9.
English, Portuguese, Italian, Spanish, French, Simplified Chinese, Japanese, and Korean. You can find the codes for these here.
I can tell you that the language code for Spanish is ‘es’
To send that to SQLcl, we’re going to use a session environment variable.
So in Windows –
c:\SQLDev\sqlcl\20.2\sqlcl\bin>SET JAVA_TOOL_OPTIONS=-Duser.language=es
c:\SQLDev\sqlcl\20.2\sqlcl\bin>sql hr/oracle
Picked up JAVA_TOOL_OPTIONS: -Duser.language=es
SQLcl: Versi≤n 20.2 Production en mar. ago. 25 15:37:58 2020
Copyright (c) 1982, 2020, Oracle. Todos los derechos reservados.
Last Successful login time: Mar Ago 25 2020 15:38:01 -04:00
Conectado a:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Se ha encontrado login.sql en el CWD. El acceso a la DB estß restringido para login.sql.
Ajuste SQLPATH para incluir la ruta y activar la funcionalidad completa.
SQL>
Error messages will come back in Spanish as well.
Mi espanol is muy malo.
Setting other JVM flags
Let’s say for…REASONS…you need to increase the amount of memory avilable to SQLcl versus what you get by default.
JVisualVM shows me what’s going on for any Java program running on my machine. I can also use it to see any JVM arguments that have been set.
Show Java does just that. I’ve also got Java VisualVM to show me what’s what for my program.
Ok, now let’s startup SQLcl again, but this time with
c:\SQLDev\sqlcl\20.2\sqlcl\bin>SET JAVA_TOOL_OPTIONS=-Duser.language=fr -Xmx800m
c:\SQLDev\sqlcl\20.2\sqlcl\bin>sql hr/oracle
Picked up JAVA_TOOL_OPTIONS: -Duser.language=fr -Xmx800m
SQLcl : version 20.2 Production sur mar. ao√t 25 15:57:21 2020
Copyright (c) 1982, 2020, Oracle. Tous droits rΘservΘs....
We can see both -Duser.language and the -Xmx flags have been recognized.
I’m not saying SQLcl needs 800MB of memory, but in case you did…
Yes, this should be documented.
We have an awesome documentation team. They’ll be adding this content ASAP in the SQLcl Quick Reference Guide.
Our biggest update since SQL Developer Web originally shipped, version 20.3 brings you:
Object Viewers
REST Development
JSON / SODA interface
See table, click and view table. Hit Esc to close the ‘Slider’Develop and manage your RESTful Web Services and OAUTH2 Clients…in your browser!
We didn’t have time to include the AUTO REST tables, views, and PL/SQL features – but that’s coming next (20.4).
Create collections, add documents, Query by Example, diagrammed data guides
For the JSON feature, you need the SODA_APP role and you need a connection a 19c or higher version of Oracle Database, then the JSON card will appear on the SQL Developer Web home page.
I have a lot more to say/show/demo on all of these features, but trust me when I say this is a BIG release for ORDS.
How easy is it to get SQL Developer Web going? Here, watch the movie!
Let’s install ORDS, & login with SQL Developer Web & publish our first RESTful Web Service!
Generate DDL (or reports) from your SQL Developer Data Modeler designs!
What about SQL Developer?
No version 20.3, although we have version 20.4 in the works, and version 20.4 of all the products are scheduled to be released in time for the end of the year.
Oracle SQLcl, your modern command line interface for Oracle Database, was released yesterday.
Lots of bug fixes, and a few new commands!
SQLcl Highlights
SQLcl – simple 30mb download and go!
I want to again, thank again the community for helping us catch bugs promptly, AND politely!
One example, there are no longer extra line breaks for statement feedback. This was very obvious when you had creating login.sql scripts. But enough about the bugs, let’s get onto the new stuff!
The primary focus for 20.3 was support for the Oracle Cloud (OCI) and specifically, the Object Storage (OSS). OSS allows you to stow files in a storage ‘bucket’ – you can then refer to these files using the OCI REST APIs, or when using DBMS_CLOUD to load tables in your Autonomous Database.
In SQLcl version 20.3, you can set a ‘cs’ session parameter to define a OSS namespace, bucket, or object. Namespaces allow you to group buckets, which you can think of as directories, and in those buckets you can have objects, which you can think of as files.
Namespaces and buckets allow you to define access permissions across many different objects to different users in your tenancy. (Oracle Docs)
Think of ‘cd’ when you want SQLcl to read or write to files in a specific directory. Now you can use ‘cs’ when you want to read from a specific place in OSS.
SQLcl – show me what’s in my bucket.
The ‘list’ command is either listo (Objects), listb (Buckets), or listc (Compartments).
SQLcl – show me what’s in my file.
Since ‘cs’ is defined down to the bucket level, I can use the ‘peek’ command, and point it to an object (/o/) in my transfer bucket (/b/)
SQLcl – get that data and use it to load a local table!
What’s my delimiter, what’s my end-of-record ‘terminator’ – load the file!
I’m connected to a local Oracle database, so I’m pulling the records from the Cloud to load into on my on-premises table, but I could load that data really to any Oracle Database I could manage to connect to from SQLcl.
What’s this mean?
I can now very easily take advantage of things I have in my Oracle Object Store from within SQLcl. To take advantage of this feature, we need to have a OCI Profile defined.
You’re going to want to CAREFULLY follow these Docs, but basically…
Create a RSA key pair in PEM format on your machine where SQLcl will be running
Upload that key to your OCI User’s profile, and get the fingerprint
Get your tenancy and user OCID, create a config file
My config file (located in your $HOME/.oci directory) looks like this –
When generating your key, don’t include a pass phrase.
Make sure your key files are locked down – ONLY your OS user should have RW rights on it. If you are too promiscuous with your key files, it will NOT work.
Once you have this done, you only need to configure your SQLcl instance to use the correct config [profile] – as you can have multiple Cloud Accounts, it’s quite likely you’ll need to create and manage multiple profiles in your config file.
The ‘oci’ command isn’t new for 20.3, but the ‘cs’ command is.
To make all of this work, and work nicely, a TON of work was done to the LOAD command, and some SET parameters.
set loadformat - control how the data will be read/interpreted set load - control how the data will be loaded
You can use the HELP command to get very nice descriptions of these settings, plus examples.
More on the CS command.
What you can do with the the CS command all depends on what the URL has been configured to. Hint, you can use ‘show cs’ to see what the active cloud storage URL is.
If the URL includes the bucket, you can just ask for objects. If it just includes a namespace, you have to include the bucket. If it includes the object, you can ONLY interact with that file.
OCI command isn’t new…call OCI REST APIs!
Let’s say I want to spool a file, and copy it up to OSS. And then when it’s up there, I want to ‘cat’ it back out.
I PUT the file to my CS URL (which id defined out to the bucket), as a locations.txt object.
The peak command automatically grabs the first 2,000 characters. If you only want the first 500…
You can additionally provide an offset, but you’ll get the first 2,000 characters in the file by default
Just one more thing…
Yes Columbo, I know this has been a long post already.
There’s so much more to talk about here, but I did mention a Data Modeler update. Real quick, in SQLcl still – there’s a new modeler command.
You can now from the CLI generate DDL scripts and reports from your existing Data Modeler Designs.
There are a TON of parameters you can feed these commands. No worries, the help includes examples!
Yesterday, Brian and I spent an hour or so talking and demonstrating all of the different options our database tools offer for loading data into the Oracle Database, especially when those databases happen to be in the Oracle Cloud (OCI).
Here is that talk:
Brian talks Autonomous, then I talk tools, then Brian demos tools.
Here are the slides:
Pretty, right?
The Referenced Resources
We referenced examples and how-to’s, those are hard to share in video and slide format, so if you’re too lazy to ‘Google,’ I did it for you
In Autonomous you already have those requirements taken care of. ALL you need is to grab the driver, which you can get here.
With that JAR downloaded, copy it to your SQLcl/lib directory.
Just paste it in there, no need to even restart your SQLcl instance if you already have it running.
And you’re ready to connect!
Autonomous Example (admin)
All of your Oracle Autonomous Database cloud services (Data Warehouse, Transaction Processing, JSON, …) will have an ADMIN user. It will be REST enabled out-of-the-box so you can login with SQL Developer Web.
Using your ADMIN database user password, you can now login.
The JDBC URL is user/password@jdbc:oracle:orest:@https://ORDSHost…/ords/USER/
My Always Free Transaction Processing Service base URL is the same I would use to login to SQL Developer Web or access a RESTful Web Service.
What’s Different?
Quite a bit. We’re no longer going to be in a transaction world where I can roll things back, or even COMMIT when I want to. We’re in the HTTPS/REST world – so think, AUTOCOMMIT.
Every request or command we make – is going to be its own transaction, even running in its own database connection. So if you’re going to use this for benchmarking, it will probably be slower than using the regular JDBC driver.
Since we’re using the Autonomous managed ORDS to connect, we’re on the ‘low’ service by default
REST Enabled schemas can be used to login to SQLDev Web & publish RESTful Web Services.
You don’t have to use SQLcl, either
It’s a JDBC driver. SQLcl is just our vendor provided utility that happens to support it. Theoretically you could use this in any of your Java applications wanting to connect to an Oracle Database!
We hosted a webinar with more than 300 Oracle developers worldwide to discuss how one could implement DevOps.
Database and APEX developers must evolve to embrace CI/CD pipelines for developing, testing, and deploying database code changes.
Easier said than done, right Developers can struggle with the concept of using a central repository for storing, reviewing, and deploying code…especially when you’re coming from a world where , all it took was a ZIP file of scripts and a DBA with SQL*Plus.
These slides & video are the work of our product manager, Brian Spendolini, aka @btspendo. The content covers how using Oracle’s free development tools combined with open source repository tools, one can create a full CI/CD pipeline for database and APEX development groups.
CI/CD Presentation & Demo
This is a pretty long session, but see the comments for Chapters and easy navigation.
CI/CD Slides
Something isn’t real if there aren’t ppt slides for it…so here they are.
The Code! CI/CD Resources on GitHub
This is a work in progress, expect more to come in the following days, weeks.
The next update for SQLcl will have some seriously cool upgrades on the overall editing experience. I’ve been teasing both vi and emacs support for example, we just couldn’t sneak those in for 21.1…
SQL Developer Data Modeler 21.1 has about a half dozen bug fixes reported from the community, mostly around DDL generation. You can see the full list here.
What about SQL Developer & ORDS?
Why, thanks for asking!
Our next update for SQL Developer will be version 21.2, around the end of June.
Oracle REST Data Services version 21.1 is ready to be released now, but we’re waiting for one last set of release procedures to clear. That update has already been submitted to Oracle Cloud for Autonomous Shared subscribers.
I’ll have a few nice updates for Database Actions, formerly known as SQL Developer Web, v21.1 hopefully in a week or 2.
wrote the first lines of SQL Developer, SQLcl, and ORDS
built a lot of the SQL workshop in APEX
leads the Database Tools team as our Vice President
responsible for hiring me (they can’t all be winners)
Kris also likes to take customer calls and present at conferences. In his spare time, when he’s not busy managing projects, people, and writing code – he puts together some very useful blogs!
In fact, about 5% of my time here at Oracle is spent directing people to a blog that Kris wrote.
So, in no particular order, here are some things you should probably know:
By default SQL Developer reads the application settings from either the Windows AppData folder, under roaming profiles, or on Linux/OS X from a .sqldeveloper directory under your $HOME.
What you can achieve with a simple JVM flag, is to tell SQL Developer to ALSO write/read those settings from someplace else…even somewhere on the innerwebs. So that if you’re using SQL Dev from 3 or 4 different machines, you’re always going to have the same SQL History, connections, etc.
This post is helpful in a few different ways. One, it’s about BLOBs, and that’s a popular topic. But more importantly, it shows how to take advantage of your local JavaScript code in SQLcl to do database stuff.
An important note regarding Java and the Nashorn engine, that’s being phased out, and in fact it’s completely gone if you’re using Java 15. So you’ll need to switch over to GraalVM – more news on this front later this year.
Making HTTPS requests need to be as fast as possible. Latency kills applications, esp when the internet is involved. So wouldn’t it be nice if your mid-tier could simply let your application know that the local data you already retrieved is still valid? That’s right, you don’t need to GET that data anymore, the data you have is already ‘good.’
That’s what this post is all about.
Obviously, your ETag Query needs to be REALLY fast, or you’re just adding new bottlenecks.
ORDS Standalone has it’s own built in webserver, or actually, we have included the Jetty project from Eclipse. In this post, Kris shows how to enable/configure something on the Jetty side of ORDS.
More fun with JavaScript. I could have easily picked a half dozen more of these, but for some reason folks like Ctrl+Alt+Delete on their database transactions/sessions/processes.
The other cool, key pickup from this post is that Kris is showing you how to add your OWN custom commands to SQLcl!
We had a great webcast today! Approximately 250 of our fellow shipmates joined us on a combined database tools retrospective and a look at where we’re going with SQL Developer, SQL Developer Web (Database Actions via ORDS), and SQLcl!
Here is the recording:
To boldly go, where SQL*Plus never did.
Here are the slides:
The cool ST fonts didn’t come through…I suspect problems in the Jeffries tubes.
Can I use Database Actions on my database, not in the Cloud? YES! Just download ORDS, and go!
Will we have full design features in the web (data modeler?) Not in the near future – for the time being you’ll want to use SQL Developer Data Modeler, unless you’re just wanting to generate data dictionary reports or relational diagrams.
What about ExaCS support for SQL Developer Web? SQL Worksheet will be super easy to launch for any Oracle DB, including ExaCS later this Summer in the Oracle Cloud.
These are both MAJOR updates, so I will just briefly summarize what you can expect in each product. Expect follow-up posts with deep-dive descriptions of the new features.
ORDS
Licensing
The biggest update for ORDS is it has moved to the Oracle Free Use Terms and Conditions license. Like SQLcl, you can now download this without registering an account or clicking through an EULA. It also means you can always find the latest version of it using the ords-latest link.
You’ll find both SQLcl and ORDS in the public YUM repo SOON.
Features
There are technical updates for both the underlying product, and some of it’s main features like Database Actions (formerly known as SQL Developer Web).
For ORDS itself, the biggest technical update is support for caching the definition of your REST Services and the privileges securing them.
What does this mean?
By default, on each request, ORDS runs a query to the ORDS_METADATA schema to figure out:
what is the SQL or PL/SQL tied to this request
what if any ORDS privileges are required to access this endpoint
These are both FAST queries. But, if you’re calling an API repeatedly, small fractions of seconds can add up quite dramatically over time.
How much?
Well, our performance minded VP and developer extraordinaire sees as much as a 5x improvement in performance if you decide to activate the caching feature.
New feature coming in @OracleREST to cache modules/rest source/privs. Local testing is 5x faster.
We are working on some officially recommended deployment architectures WITH some performance benchmarks attached so you can see what you can expect with ORDS on various hardware platforms/specs, stay tuned for that later this Summer!
If you enable the cache, you need to be aware that any changes to your REST APIs may take as long as the lifetime of the cache to be active in your environment. If you need the change to be immediate, you would need to restart ORDS.
Again, this is OFF by default.
ORDS/Database Actions/Charting
Running queries and seeing the results in a spreadsheet-like view is nice. But sometimes pictures tell the story better. And sometimes you want to use these pictures, over and over.
So Database Actions is adding a new feature, Charts. It’s accessible via the menu or by its card on the ‘Launchpad’ / home screen.
We like simple, self-explanatory feature names
This is the ‘direct’ route to the feature. In this workflow, you’ll give us a SQL statement, then we’ll try to apply a visualization to that data given at least 1 label and 1 number attribute.
A polar-coordinate area chart.
This data…fed by a SQL statement.
You then pick the columns you want to feed the X and Y and…you’ve done charts before I’m guessing.
Bigly data? Group/SORT the stats w/SQL vs asking the browser to take your thousands of data points into account…
Like I said, much much more coming on this topic. I will say today that this is a Part 1 of the planned feature. Part 2 is the ability to build dashboards that are composed of multiple charts.
Ok, onto our favorite command line interface for the database
ORDS / Database Actions / JSON
For your SODA/JSON collections, we now give you some really nice wizards for building both indexes and relational views. This can make accessing your data both EASIER & FASTER.
Point and click your way through your JSON attributes to decide what needs Indexes or used for a VIEW.
There’s actually more than this to look forward to in Database Actions, but again, we’ll get more into those details in follow-up, deep-dive posts.
SQLcl
This is a big, big update to a major core component of the product: the editor and prompt display. The inline editor now supports BOTH vi and emacs.
And, the annoying things where you would invoke sql recall or the tab complete, and the cursor would ‘jump’ to the bottom of the screen? That’s not going to happen anymore.
And, we’ve added an optional statusbar!
Use ‘set statusbar on’ and ‘help set statusbar’ to see how this works.
An idea with what you can do with the vi support –
You don’t need to toggle into ‘vi’ anymore, that’s the default.
The syntax highlighting you see in the GIF above is also configurable…
Again, more details on this coming soon if you don’t want to step into the help.
What about SQL Developer & Data Modeler?
Both products are getting updates with plenty of bug fixes…hopefully next week. We’ll of course let you know all about that when they’re released.
With version 21.2 of SQLcl, you have some new, exciting toys to play with! Now, some of you may have heard via the grapevine that the big thing coming to SQLcl was built-in support for vi and emacs when writing or editing your SQL and PL/SQL code.
But before we cover that, I want to cover two additional features.
One, the ability to apply color coding to your text based on our SQL parser.
And two, there’s a new status display bar you can use to see if you have pending transactions, who you’re logged in as, how long your last query took to ran, etc.
Let me show you how that might appear before I show you how to set it up.
As I type, the editor updates the coloring…this might be a bit jarring if you’re a slow typer.
So I’ve got black text with white background comments, green keywords, yellow strings, purple identifiers – thanks Dracula theme makers for the inspiration!
Then at the bottom of my screen I have something telling me my cursor line/char pos, who I’m logged in, the service I’m logged into, my editor mode, my transaction status, and the run time of that last query.
Here are the commands you will need to investigate:
set statusbar
set highlighting
help set statusbar
help set highlighting
Use SET to turn things on or change the configuration, use the ‘help’ to get instructions and examples for doing said SETs.
I’m not going to PRINT that help text here – it’s going to be changing as new versions come out anyway – but I WILL show you my login.sql, so you can use it as a quick start.
Before that, a few more things to know –
you can define the ordering of the items shown in the status bar
the status bar doesn’t currently, continuously update the statuses
the status bar doesn’t currently allow you to add your own custom status items
these things are off by default – you must turn them on
My login.sql to setup highlighting and status bar
set statusbar on
set statusbar add editmode
set statusbar add txn
set statusbar add timing
set highlighting on
set highlighting keyword foreground green
set highlighting identifier foreground magenta
set highlighting string foreground yellow
set highlighting number foreground cyan
set highlighting comment background white
set highlighting comment foreground black
What this looks like, in real time…
Yes, I type real slow like…more like I didn’t know what I was going type until I typed it.
You’ll notice the status bar was changing as I was building the SQL to show the current curpos by line:character number.
Let’s talk about the status bar some more…or better yet let’s demo!
Keep your eye on the status bar items…
So after our ‘big insert’ we have ‘Changes Pending.’
But, as soon as our INDEX is created, it goes back to ‘None.’
Why? Because executing DDL is an implicit COMMIT.
Did you also notice that the text on screen and cursor didn’t start jumping around the screen as I invoked the command history (with my up arrow key?) We’ve FIXED that behavior in this release as well!
This DDL/DML scenario courtesy our AskTom hero, Connor. He’s been helping me in his spare time to ‘beta test’ SQLcl’s new editor features. Thanks Connor!
Wait, wait…what about Windows?
It works/looks exactly the same in your Windows CMD prompt! Our developer on this feature uses Windows by default, and I of course have a Windows corp machine where I do MOST of my work. Everything I’ve shown on my Mac here should appear the same in your MSFT universe.
Subscribers and my mom will probably remember that I’ve briefly talked about this feature before, but it was really just a tease. I wanted to go into a bit more detail today.
In version 21.3, SQLcl got a wicked cool new feature. And yes, it sounds pandering of me to say that, but every now and then I see new features come out that turn out to be JUST as handy as we imagined them when we set out to build them.
This is one of those feature I’ll probably toss out in every Tips & Tricks talk I do going forward.
You have: a delimited text file or simply a CSV.
This is my personal data dump from the social media app known as Untappd.
But, it’s a wizard, has multiple steps, and I don’t have SQLDev started, and I’m already at my prompt, ready to go, NOW.
The New-School Way
The LOAD command has a new parameter you can toss onto a job, ‘NEW’.
This looks promising!
So what does this command do? Well, we scan your data from the CSV, we look at the column headers to come up with new column names, and then we look at the data itself – how wide are the strings, is that a DATE format we recognize, etc.
Then we show you that DDL, execute it, and then load the data from the CSV into the new table.
Which looks a LITTLE something like this –
The output keeps going…this is just the first bit showing me the load options I have going.
The most interesting thing of note here is this:
scan_rows 1000
That’s a LOAD command option telling SQLcl to look at the first 1,000 rows of my CSV to ‘measure’ the column widths and dates to build the DDL/INSERTs around.
If you have wider data past the first 50 or 100 rows, you’ll get a lot of REJECTED INSERTs.
SET LOAD SCAN_ROWS 1000
Note the higher you set this, the more resources you’ll burn reading the data and doing the number crunching.
So, I don’t need to do anything really, I can just toss my CSV at SQLcl, and let it put it into a table for me.
Here’s a quick animation…
Create and Load the table, collect stats, INFO+, and a little reporting query of my new data to play with.
Or maybe I just want the propose table DDL…
SHOW DDL, what’s that?
So go through the 1000 rows, and figure out the DDL, but just SHOW it to me, don’t execute it.
load newtable file.csv SHOW_DDL
If you add the ‘NEW’ onto the command, it will actually execute the scenario.
A brand new Oracle Cloud (OCI) Service launched yesterday, and this post will describe briefly:
what it is
why we built it
what you can do with it today
what we have planned for the future.
First things first – everything here is free. Defining connections is free. Storing secrets (credentials and passwords) in the Vault is free. Creating bastions to SSH into private endpoints is FREE. This service is all about making your databases easier to access, either directly using our tools or via your apps.
The Database Tools Service isn’t a new way to create, run,or manage databases on OCI, it’s for WORKING WITH those databases.
You can create many different types of Oracle databases on OCI – Autonomous, Exadata CS, VM based instances, or even do up your own install on a Compute node.
When you’re ready to use those databases, what’s the first thing you might want to do? I can think of a couple of big ones:
connect to it
run some queries.
Connections
What is a connection, really? It’s everything you need to know to connect to a database. It’s going to have the location of the database, it’s name, listener port, and a username/password. With that information, you can create a connection in your tools and programs.
So if we were to visualize this say your favorite desktop IDE for Oracle, that could look like this –
So for SQL Developer, a Java application, Advanced panel lets you ‘inject’ JDBC connection properties.
As a client application, these are stored on the machine with your program. Which is nice, if you’re on that machine, but it’s 2021, and we’re in the Cloud, and there should be an easier way to deal with our critical resources, like Oracle Databases!
Cloud also introduces questions. Like, how can we connect to our database without making it ‘open’ to the entire Internet.
Most databases in OCI use private subnets (for good reason!), and that poses a challenge when connecting to them. This service makes creating bastions and managing private endpoints MUCH easier for application developers, or people using database tools to connect to those database.
This post will show an example of an ‘open’ database and connecting to it directly.
A Quick Demo to give you an idea of what’s possible.
I can create a connection, stow my password in the OCI Vault, and then use that information to say launch SQLcl to my Autonomous Database, on-the-fly, without doing anything other than clicking a single button.
And don’t forget, Autonomous Databases classically require Client Credentials containing your two way TLS encryption keys, so that the database can trust us as a client. Managing these ‘wallets’ takes a bit of work, even if we’ve coded a lot of that away for you with our tools.
My connection resource has already been created in OCI – now I can use it whenever I want.
This GIF if pretty short, and a lot happens in a small amount of time, but what you’re seeing is me asking for a SQLcl session to my Autonomous Database, defined in my Connection resource.
Which results in SQL GETTING:
the database location, name, and particulars
the TLS Client Credentials
the database username and password
the database connection…which leads to me being…
…in my DB, authenticated and ready to go!
Beyond the scope and nature of a ‘Hello World!’ post, if your database is also accessed via a private endpoint, the SQLcl can also use that information to create an SSH tunnel/Bastion to make the connection as well!
Another Quick Demo
One of the bigger things our team has put together over the past few years is SQL Developer Web. This is surfaced in the Autonomous Database Services as ‘Database Actions.’ And it’s great. But.
But.
You have to go to the database, and THEN launch Database Actions, connect and start your work. Hint: You can bookmark each Autonomous instance to go straight to the connect bits…but it’s always one link, one page, for ONE database.
What we want you to be able to do is, launch your SQL scratchpad, choose your database, run your queries.
Like this:
I can switch to any connection I have defined in my tenancy.
Let’s Build a Connection!
This is going to be your Day 0, Minute 0 project to get started using the Service. Let’s cover some housekeeping and ground rules.
To build a connection, we’re going to go to that first link, ‘Connections.’
What happens probably if you just jump in without doing your homework…
You’ve been working with databases a long time, no need to ready any docs, yeah?
Well, when it comes to this part of defining your connection, if you don’t already have at least one Vault available…
Can’t click ‘Create’ until I have a Vault and Encryption Key defined.
We need some place to put the password(s). That ‘place’ is the OCI Vault. Or more succinctly:
Oracle Cloud Infrastructure Vault is a managed service that lets you centrally manage the encryption keys that protect your data and the secret credentials that you use to securely access resources. Vaults securely store master encryption keys and secrets that you might otherwise store in configuration files or in code. Specifically, depending on the protection mode, keys are either stored on the server or they are stored on highly available and durable hardware security modules (HSM) that meet Federal Information Processing Standards (FIPS) 140-2 Security Level 3 security certification.
And to access the things IN your Vault, you also need a Key. Now, these requirements and others are covered in the Database Tools docs – so do your homework and read those. And, we’d like to make this even easier going forward, so stay tuned.
The good news is, creating a vaule is probably something you need to do just once per project/application/group of ‘things’ you want to manage.
It literally took me about 90 seconds to setup my Vault and Key…
Defining the Connection
Each connection will have it’s own username and password. It will be for a single database – be that a PDB, CDB, or classic single instance architecture database (11gR2..21c). And it can be for the very cheapest resource, Always Free, up to the most very expensive shape we offer. These services are all included and free to use with your OCI subscription.
Remember our connection in SQL Developer? Most of this should seem very familiar.
Now, we do offer some very nice touch-features. Choose your type of database –
If you choose ‘Enter Database Information’ you can manually point to any instance on your network!
Here’s where the convenient part comes into play.
Everything known to the OCI Database REST APIs
We can query the OCI control plane to get a list of your Autonomous Database services. And with those, comes the information we need to get the Client Credentials. Just point, click, and you’re done.
Once you fill everything out on the first page…
I have my Vault and Key already defined, so I’m ready to ‘stow’ my ADMIN password. That step isn’t shown but it’s just a classic ‘put in the password TWICE’ to make sure it’s right input form.
…just click Next.
We’re ALMOST done.
Since I’m defining a connection to an Autonomous Shared Infrastructure Database, I MAY need to provide credentials to enable mutual TLS (mTLS).
This is NOT the Secret for your database user password in the Vault!
If you accidently choose the same secret you supplied previously for your database user’s password, you’ll see this:
There’s Zero chance these secrets would work for both your DB user and your mTLS credentials.
Getting Confused, Worried? Don’t be, this is still Easy!
Once you click that ‘Create Wallet Content Secret’ button, you’ll see this –
9x out of 10, you’ll just use the defaults here for ‘Retrieve auto login wallet’
We’re smart enough to just use the Autonomous Database Service APIs to go ask it for the credentials FOR YOU. Now, if you already downloaded your wallet, or if you want to specify exactly what’s being used, just use the ‘Upload’ option.
That’s it, you’re ready! Just hit the ‘Create’ button.
You’ll be back on the Connections screen, and you can start playing around with said connections.
Hamburger button on each Connection item will allow me to execute some calls.
If I open a Connection, I can see it’s treated like any other Top Level OCI Resource – it has an OCI and everything – it can be tagged for easier filtering, or more importantly, you can restrict access to it from your fellow OCI users/apps via Group roles, etc.
Remember the Advanced tab in SQL Developer? You can put those same properties here at the bottom.
The two biggest features today are highlighted first – SQL Worksheet and Launch SQLcl. The Worksheet will open with this Connection active by default. But, I could go to the Worksheet directly and then pick a connection, it’s really up to me, and where I’m at in the console.
One nice ‘debug’ feature is the Validate command –
We can reach the database, and we can make a connection with the credentials stowed in the Vault.
I’ll cover ‘debugging’ connections in a subsequent post, but this can help you figure out if you got something wrong or if your password has expired, or…
Is this really all it is?
I mean, it would still be cool if THIS was all it was. But, it’s not. It’s really only HALF the story. You see, since it’s an OCI resource, we can use the OCI SDKs, APIS, etc to use these resources.
So, if I had a Python program that needed to connect to a database, it could simply call the Database Tools service to ‘Get’ the connection, bring it down, construct the connection object, and make the connection – all without the developer knowing the details except for the OCIDs.
For starters the SQL Worksheet is extremely basic. I’ll cover it in a future post, but it doesn’t come close to offering the full power of SQL Developer Web. Porting that Oracle JET app so it can run natively in the OCI Console is an ongoing project and I expect huge progress in 2021.
Also, this isn’t just for us. It’s for any other Database related service in OCI that needs to work with databases. We’ll be at their disposal too, not just yours.
And it’s not necessarily exclusive to Oracle Databases. OCI supports other types of databases, so we’ll be including support for all database application developers in the Oracle Cloud.
And finally, it would be nice if we could do things like say ‘Install ORDS Metadata’ or ‘Upgrade APEX’ for a known Oracle database via a Connection.
The roadmap here is long and filled with many scenic views, so we’re very excited to get there. But, most importantly, we’re excited to get this Service off the ground and running!
Thanks to everyone on all the teams that made this launch possible. Expect many updates in the form of slides, videos, blog posts, Github How-To code examples, podcasts, and more.
The Movie
Finally, we can’t release stuff without having a video! Brian and I talk about and demo connections and our tools. See the video description on YouTube for chapters you can use to jump to specific topics.
I recently put together a 30 minute ‘power session’ for our Oracle Database World event. If you missed it, you still have one opportunity to attend the simu-live event on December 1-2. We’ll be there live to answer your questions and take you on a guided tour of our hands-on LiveLabs.
If you’d like to see my Tips & Tricks session, you can watch that now.
Yes, I also hate my face. Press play QUICKLY to make it go away.
This Tips & Tricks session doesn’t just cover SQL Developer and SQLcl. It ALSO covers SQL Developer Web, ORDS, and SQL Developer Data Modeler.
AND, it includes a bonus tip from Maria Colgan!
Q&A from the Americas and European Events
Is there a prebuilt CLI available for calling the OCI ORDS Server for doing this i.e. creating the Oauth clients?
You can invoke the OAUTH PL/SQL API from SQLcl…or use the OAUTH2 GUI in SQL Developer Web to manage your Clients.
How would I find the URL for the SQL Developer Web if the admin would not give us any information (all we have is the wallet to connect to the ATP) and we use that with our IDE to connect to the Cloud
Ask your admin, very nicely to share it. Otherwise it sounds like you’re asking how to hack into your system, which I obviously can’t help with.
If we use ORDS With OAuth for the web tier, is the DB connection via a shared user/schema in which case how can we identify the end user inside the database?
You can query :current_user within the handler code block which will give you the client ID – I talk about that here.
The developers are not given access to OCI console (Database tools). The only we can access the ATP is from our IDE. How do we REST enable?
REST Enabling is available in SQL Developer desktop via a simple right-click, or you can invoke the ORDS PL/SQL APIs manually in a script or EXEC command.
Can you use custom domain with oracle ADB rest api?
Yes, absolutely.
We use ATP, and if I am going to try out REST on packages & procedures, am I going to start any new services and affect any performance (and upset DBAs)?
Only if your services are poorly coded in terms of the SQL and PL/SQL you use to build those services.
There are a lot of DB side features for data redaction/restriction meaning BOB and ALICE might see different results on the same query. Are an ORDS/REST applications able to still use these?
Yes! The REST APIs run under a schema, any security rules written for BOB or ALICE will apply the same regardless if invoked via HTTPS or via SQLcl/SQLDev.
Do you need any special privilege to REST enable?
You need to own an object, or you need the ORDS_ADMIN role in the database.
what if the PLSQL procedure has 50 parameters? Would you build a big query string? And what about if there is types as parameter (structure)?
Types are supported and you’d have a JSON payload on the POST request. If you want to pass parameters on the query string, you could have 50, but it would be ugly.
Does REST enabling your app meaning moving security outside of database users/grants and if so wouldn’t that make it incompatible with something like DB Vault?
There are two layers of security, what the mid/web tiers let pass through to the database, and then what the database allows once you’re inside the database.
DB Vault as in the sense of building security realms and command rules etc. Ultimately meaning that BOB and ALICE might see different results to the same query – would it be compatible with a REST/ORDs application? Because on the DB side the REST/ORDs application might always connect as the same database user?
You can use our prehook feature to run custom pl/sql at the authentication/authorization step to set database session parameters, etc. In general database users are NOT web/actual users though.
Are there any example systemd unit file examples for running ORDS from linux?
We have RPMs for ORDS on Linux avail via YUM.
Are the “info+” elements a report of what’s really in the data? Or are they reports on what’s in the stats?
It’s reporting what’s in the database stats – we don’t query the data itself, that would be terribly expensive.
Excellent presentation, we use all of those tools, and we could never done it without all the demos
Ah, thanks that was very nice of you to say.
Is there a way to prevent ORDS autonomous from showing adb url ??( in case of use vanity url), was not able to set HTTP_ERROR_STATUS_ON_ERROR_PAGE_ENABLED
No, we manage the mid-tiers for you in Autonomous.
What version of SQL Developer is being presented?
21.2 for desktop and 21.3 for Web.
IS there a docker image available for the latest version of Oracle XE?
Which SQLDev version is recommended for most of the features shown ?
Always, the latest (21.2)
Does ORDS require weblogic or can you run ORDS in production as a FatJar?
We don’s require WLS, but if you want to use it, it’ll be just fine there. if ‘FatJar’ means run as a standalone java application, absolutely we support that for production.
Can I create a REST service on an Autonomous Database?
Yes, yes, and more yes.
Jeff the current report you are showing is present in sql dev 21.2.1 ?
They were talking about a Web Dashboard in SQL Dev Web. We do have reports and charts in SQL Dev desktop, but not quite the same as what we have in the Web.
Can you run Oracle Database Actions page on premise with an oracle 19c environment?
YES.
Can SQLcl search and edit history with VI/vim commands?
No, but I want that feature myself, so stay tuned.
How to link SQLcl with VI or Emacs?
You can control this via the ‘set editor’ command, but latest SQLcl runs with it’s own editor/vi built-in by default, or you can switch to Emacs if you’re a nerd.
Did we update the OCI Test option to work with the 19c client? — It stopped working recently
Make sure you’re on 21.2.1 – that’s been fixed to be A-OK with 21c or 19c clients.
when we cast a column in SQL to a type (with multiple columns and rows), we are not able to see the actual data in SQL Developer.
Be sure to disable this preference, you’ll avoid the issue with slow queries/excessive hits to the data dictionary:
You’ll lose a lot of features, so try to upgrade to 21.4.2!
One last thing on this log4j business
A user asked, and it’s a good question:
Thanks for the question, DD!
Yeah, you can simply delete or rename the log4j-core.jar file in the sqldeveloper/lib folder, and SQL Developer will work just fine – so long as you’re not using it for the Lifecycle Management Pack feature from OEM, as shown here:
Change plans and Oracle Change Manager.
What about SQLcl?
The primary bug fix here was also fixed for SQL Developer.
33676971 REMOVE ENV VARIABLE SUBTITUTION FROM COMMON LIBRARY
This can be bad, as $ is a valid character for Oracle object names, like… V$TEMP.
(@krisrice) 
(@krisrice) 
